DagsHub

DECEMBER 11, 2023

Dolt Created in 2019, Dolt is an open-source tool for managing SQL databases that uses version control similar to Git. It versions tables instead of files and has a SQL query interface for those tables. It provides ACID transactions, scalable metadata management, and schema enforcement to data lakes.

Machine Learning 98

Machine Learning Machine Learning Data Lakes Data Science 98

ODSC - Open Data Science

FEBRUARY 24, 2023

Great Expectations GitHub | Website Great Expectations (GX) helps data teams build a shared understanding of their data through quality testing, documentation, and profiling. With Great Expectations , data teams can express what they “expect” from their data using simple assertions.

Exploratory Data Analysis 126

Exploratory Data Analysis Data Visualization Data Analysis Data Analysis 126

Pickl AI

APRIL 21, 2025

Semi-Structured Data: Data that has some organizational properties but doesn’t fit a rigid database structure (like emails, XML files, or JSON data used by websites). Unstructured Data: Data with no predefined format (like text documents, social media posts, images, audio files, videos).

Big Data 52

Big Data Big Data Data Science Machine Learning 52

phData

SEPTEMBER 26, 2023

A common problem solved by phData is the migration from an existing data platform to the Snowflake Data Cloud , in the best possible manner. Sources The sources involved could influence or determine the options available for the data ingestion tool(s). These could include other databases, data lakes, SaaS applications (e.g.

SQL 52

SQL Database ETL Data Models 52

AWS Machine Learning Blog

MAY 31, 2024

Challenges and considerations with RAG architectures Typical RAG architecture at a high level involves three stages: Source data pre-processing Generating embeddings using an embedding LLM Storing the embeddings in a vector store. Vector embeddings include the numeric representations of text data within your documents.

AWS 133

AWS Machine Learning Machine Learning Database 133

The MLOps Blog

JUNE 27, 2023

User support arrangements Consider the availability and quality of support from the provider or vendor, including documentation, tutorials, forums, customer service, etc. Check out the Kubeflow documentation. Metaflow Metaflow helps data scientists and machine learning engineers build, manage, and deploy data science projects.

Machine Learning 134

Machine Learning Machine Learning ML ML 134

DataRobot Blog

MARCH 7, 2023

The DataRobot AI Platform seamlessly integrates with Azure cloud services, including Azure Machine Learning, Azure Data Lake Storage Gen 2 (ADLS), Azure Synapse Analytics, and Azure SQL database. This drastically improves productivity of teams and allows them to scale business results.

Azure 59

Azure Machine Learning Machine Learning AI 59

phData

APRIL 29, 2024

Oracle – The Oracle connector, a database-type connector, enables real-time data transfer of large volumes of data from on-premises or cloud sources to the destination of choice, such as a cloud data lake or data warehouse.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

DagsHub

OCTOBER 23, 2024

Here’s the structured equivalent of this same data in tabular form: With structured data, you can use query languages like SQL to extract and interpret information. In contrast, such traditional query languages struggle to interpret unstructured data. Storage Tools To work with unstructured data, you need to store it.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

The MLOps Blog

OCTOBER 20, 2023

References : Links to internal or external documentation with background information or specific information used within the analysis presented in the notebook. Data to explore: Outline the tables or datasets you’re exploring/analyzing and reference their sources or link their data catalog entries. documentation.

SQL 52

SQL Database Data Scientist Python 52

Alation

JULY 19, 2022

For example, a new data scientist who is curious about which customers are most likely to be repeat buyers, might search for customer data only to discover an article documenting a previous project that answered their exact question. Query editors embedded directly into data catalogs have a few advantages for data scientists.

Data Scientist 59

Data Scientist Data Quality Data Science Data Analyst 59

Pickl AI

APRIL 9, 2025

This includes operations like data validation, data cleansing, data aggregation, and data normalization. The goal is to ensure that the data is consistent and ready for analysis. Loading : Storing the transformed data in a target system like a data warehouse, data lake, or even a database.

ETL 52

ETL Data Warehouse AWS Business Intelligence 52

IBM Journey to AI blog

JULY 17, 2023

These encoder-only architecture models are fast and effective for many enterprise NLP tasks, such as classifying customer feedback and extracting information from large documents. While they require task-specific labeled data for fine tuning, they also offer clients the best cost performance trade-off for non-generative use cases.

AI 74

AI AI Machine Learning Machine Learning 74

Alation

SEPTEMBER 23, 2021

Active Governance – Active data governance creates usage-based assignments, which prioritize and delegate curation duties. It also allows for deeper analytics and visibility into people, data, and documentation. In this sense, native integration capability is the bare minimum requirement to connect to data sources.

Data Preparation 52

Data Preparation SQL Data Governance Data Analysis 52

phData

DECEMBER 4, 2023

For greater detail, see the Snowflake documentation. If you answer “yes” to any of these questions, you will need cloud storage, such as Amazon AWS’s S3, Azure Data Lake Storage or GCP’s Google Storage. Loading small amounts of data is cumbersome and costly: Each insert is slow — and time is credits.

Database 98

Database Clustering SQL Data Pipeline 98

AWS Machine Learning Blog

JUNE 25, 2024

In addition, the generative business intelligence (BI) capabilities of QuickSight allow you to ask questions about customer feedback using natural language, without the need to write SQL queries or learn a BI tool. The following diagram illustrates the architecture and workflow of the proposed solution.

AWS 129

AWS Natural Language Processing Machine Learning Machine Learning 129

Pickl AI

JANUARY 28, 2025

External Data Sources: These can be market research data, social media feeds, or third-party databases that provide additional insights. Data can be structured (e.g., documents and images). The diversity of data sources allows organizations to create a comprehensive view of their operations and market conditions.

Business Intelligence 52

Business Intelligence Business Intelligence ETL Data Lakes 52

Mlearning.ai

JULY 10, 2023

There are other options you can place, and as usual, I suggest you to reference the official documentation to learn more. To create a Scheduled Query, the initial step is to ensure your SQL is accurately entered in the Query Editor. These functions can then be used in your SQL queries in BQ to simplify and optimize your analysis.

SQL 52

SQL Database Database Administration Data Lakes 52

AWS Machine Learning Blog

MARCH 8, 2023

Data collection and ingestion The data collection and ingestion layer connects to all upstream data sources and loads the data into the data lake. Marubeni’s internal models are based on Long Short-Term Memory (LSTM) architectures, which are well documented and easy to implement and customize in TensorFlow.

AWS 101

AWS Machine Learning Machine Learning Analytics 101

Mlearning.ai

FEBRUARY 16, 2023

Thus, the solution allows for scaling data workloads independently from one another and seamlessly handling data warehousing, data lakes , data sharing, and engineering. Data warehousing is a vital constituent of any business intelligence operation.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

IBM Journey to AI blog

JULY 3, 2024

As data types and applications evolve, you might need specialized NoSQL databases to handle diverse data structures and specific application requirements. Enterprises might also have petabytes, if not exabytes, of valuable proprietary data stored in their mainframe that needs to be unlocked for new insights and ML/AI models.

AI 45

AI AI Data Quality Database 45

Alation

APRIL 14, 2022

Data pipeline orchestration. Support for languages and SQL. Moving/integrating data in the cloud/data exploration and quality assessment. For example, data science always consumes “historical” data, and there is no guarantee that the semantics of older datasets are the same, even if their names are unchanged.

Data Governance 59

Data Governance ML ML Cloud Data 59

Pickl AI

SEPTEMBER 5, 2024

Storage Solutions: Secure and scalable storage options like Azure Blob Storage and Azure Data Lake Storage. Key features and benefits of Azure for Data Science include: Scalability: Easily scale resources up or down based on demand, ideal for handling large datasets and complex computations.

Azure 52

Azure Data Scientist Data Science Machine Learning 52

phData

JULY 10, 2023

Why External Tables are Important Data Ingestion: External tables allow you to easily load data into Snowflake from various external data sources without the need to first stage the data within Snowflake. Data Integration: Snowflake supports seamless integration with other data processing systems and data lakes.

Data Lakes 40

Data Lakes Azure Database AWS 40

phData

MAY 6, 2024

Another benefit of deterministic matching is that the process to build these identities is relatively simple, and tools your teams might already use, like SQL and dbt , can efficiently manage this process within your cloud data warehouse. Store this data in a customer data platform or data lake.

Data Lakes 52

Data Lakes Data Warehouse Cloud Data SQL 52

phData

FEBRUARY 14, 2023

Data Collector also offers replication and Change Data Capture (CDC) to be able to accurately and efficiently get your data into Snowflake. Data Collector can use Snowflake’s native Snowpipe in its pipelines.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

phData

SEPTEMBER 28, 2023

Attach a Common Data Model Folder (preview) When you create a Dataflow from a CDM folder, you can establish a connection to a table authored in the Common Data Model (CDM) format by another application. This path is essential for accessing and manipulating the CDM data within your Dataflow.

Power BI 78

Power BI Data Preparation Machine Learning Machine Learning 78

The MLOps Blog

MARCH 21, 2023

I have worked with customers where R and SQL were the first-class languages of their data science community. Solution Data lakes and warehouses are the two key components of any data pipeline. The data lake is a platform where any kind or amount of data can be stored, processed, and analyzed.

Machine Learning 69

Machine Learning Machine Learning Data Scientist ML 69

The MLOps Blog

JUNE 5, 2023

One of the hardest things about MLOps today is that a lot of data scientists aren’t native software engineers, but it may be possible to lower the bar to software engineering. I’ve seen tools that help you write and author pull requests more efficiently, and that help automate building documentation.

ML 52

ML ML Machine Learning Machine Learning 52

AWS Machine Learning Blog

JANUARY 26, 2024

Accelerate your security and AI/ML learning with best practices guidance, training, and certification AWS also curates recommendations from Best Practices for Security, Identity, & Compliance and AWS Security Documentation to help you identify ways to secure your training, development, testing, and operational environments.

AWS 139

AWS ML ML AI 139

AWS Machine Learning Blog

OCTOBER 11, 2024

Use cases for vector databases for RAG In the context of RAG architectures, the external knowledge can come from relational databases, search and document stores, or other data stores. A RAG workflow with knowledge bases has two main steps: data preprocessing and runtime execution. All these steps are managed by Amazon Bedrock.

Database 107

Database AWS Clustering Data Lakes 107

Pickl AI

FEBRUARY 23, 2025

Look for features such as scalability (the ability to handle growing datasets), performance (speed of processing), ease of use (user-friendly interfaces), integration capabilities (compatibility with existing systems), security measures (data protection features), and pricing models (licensing costs).

Big Data 52

Big Data Big Data Apache Hadoop Apache Kafka 52