Spotfire Ideas Portal
Add a new idea Subscribe
Status Implemented
Product Spotfire
Categories Data Access
Created by Guest
Created on Feb 27, 2024

RELATED IDEAS

Add connection parameter for Apache Spark SQL connector

Request

Provide input for optional connection parameters for Apache Spark SQL connector (see attached .png).


Background

Apache Spark SQL connector is used to connect Databricks into Spotfire.


Connecting to Databricks catalog and databases works as designed, but querying certain large data results in an error (see Error below).


Researching online suggests the solution is to set the parameter `EnableQueryResultDownload="0"` in driver config (see Reference below); however there's no option to do this via Spotfire connector pane.



Error

Spotfire.Dxp.Framework.Library.LibraryException: Import failed ---> Spotfire.Dxp.Data.Exceptions.ImportException: An error occurred when executing a query in the external data source.

External error:

ERROR [HY000] [Simba][Hardy] (35) Error from server: error code: '0' error message: '[Simba][Hardy] (134) File df2ca4b4-fc10-4eab-adf3-aca527a9c2bb: A retriable error occurred while attempting to download a result file from the cloud store but the retry limit had been exceeded. Error Detail: File df2ca4b4-fc10-4eab-adf3-aca527a9c2bb: The result file URL had expired on 1708645311447 (Unix timestamp)'.


Reference

Setting [Simba Config](https://community.fabric.microsoft.com/t5/Service/Error-on-PBI-Service-dataset-refresh/m-p/2633552).

apache-spark-co...
apache-spark-connection-option.png
Open full size
apache-spark-connection-option.png
Implemented in 14.4
  • ADMIN RESPONSE
    Jun 14, 2024

    From Spotfire 14.4, the web client can be used to create and modify connections to Databricks and other Apache Spark SQL based data sources. The new web UI (also available in the installed client) allows for selecting OAuth2 as authentication method (in addition to username/password and Windows authentication) for SSO and safe personalized login.

    When needed, connections can be customized by using custom connection properties for data source or driver specific settings.

    Note: Some connection authoring capabilities, such as configuring prompts, writing custom queries, and defining primary key columns, are only available when you use the installed client. For details, see the documentation.

    Learn more about everything new in Spotfire 14.4.

  • Attach files
This document (including, without limitation, any product roadmap or statement of direction data) illustrates the planned testing, release and availability dates for Spotfire products and services. It is for informational purposes only and its contents are subject to change without notice. Planning to implement - generally 6-12 months out. Likely to Implement - generally means 12-18 months out. Copyright © 2014-2023 Cloud Software Group, Inc. All Rights Reserved. Cloud Software Group, Inc. ("Company") follows the EU Standard Contractual Clauses as per the Company's Data Processing Agreement. Terms of Use | Privacy Policy | Trademarks | Patents | Contact Us