Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
363d4bf
Version change to 1.11.7
piotrczarnas Jun 20, 2025
1ed27cb
Add select all functionality for column checkboxes in EditProfilingRe…
Jun 23, 2025
e4c5a74
Merge branch 'develop' of https://dev.azure.com/imagetemplates/documa…
Jun 24, 2025
2cb8221
saving where filter correctly
Jun 25, 2025
82312ec
Handle empty column mappings. Table quality status calculated correct…
piotrczarnas Jun 25, 2025
97da03d
Include the version number in the configuration. Do not raise errors …
piotrczarnas Jun 25, 2025
4c2d128
Python calls manage MaxDOP in a more reliable way on low-CPU machines.
piotrczarnas Jun 26, 2025
be23798
Fix to compare negative values
piotrczarnas Jun 27, 2025
dbdcbaa
Silently drop additional rows when comparing data
piotrczarnas Jun 28, 2025
321620f
Python process caller uses a timeout to wait for an available process.
piotrczarnas Jun 29, 2025
aadceb7
Support downloading logs as a zip file.
piotrczarnas Jun 29, 2025
a3c96fe
Job execution logging
piotrczarnas Jun 29, 2025
41d3afe
The initial retrieval of jobs does not return jobs, only the change i…
piotrczarnas Jun 29, 2025
59a2e3e
Sort results by the time period first, not the execution time.
piotrczarnas Jul 3, 2025
c571d30
Always show the actual and expected values
piotrczarnas Jul 4, 2025
60b0d04
Hide errors raised by springfox.
piotrczarnas Jul 5, 2025
f94023a
Small fixes in the configuration
piotrczarnas Jul 5, 2025
3bf57b2
Develop build script adds closed source code
piotrczarnas Jul 5, 2025
c963b81
Fix the github action - references to secrets
piotrczarnas Jul 5, 2025
e626c89
Change the merge behavior during the github action build.
piotrczarnas Jul 5, 2025
e3ca7aa
Github action checks out the paid version despite different histories
piotrczarnas Jul 5, 2025
e6841d0
Apply additional configuration of Git in the github action
piotrczarnas Jul 5, 2025
38d816b
Add a merge strategy
piotrczarnas Jul 5, 2025
711b707
Use the "ours" merging strategy.
piotrczarnas Jul 5, 2025
18d5427
Add closed source build features to the release build script
piotrczarnas Jul 5, 2025
1feece6
Apply closed source code to rebuild
piotrczarnas Jul 5, 2025
dafc15d
Push closed source changes
piotrczarnas Jul 5, 2025
68e91b6
When merging code with the closed source edition, use the closed sour…
piotrczarnas Jul 5, 2025
9ecac14
Unique column selection popup changed to fit on the screen. Improveme…
piotrczarnas Jul 11, 2025
22faa0c
Update the version.
piotrczarnas Jul 14, 2025
1a520b0
Use the closed source github SHA to tag docker containers.
piotrczarnas Jul 15, 2025
94f58b0
Shorten the docker develop hash to 8 characters.
piotrczarnas Jul 15, 2025
2e9bd88
Change the license to BUSL 1.1
piotrczarnas Jul 21, 2025
87094da
The pom.xml license changed
piotrczarnas Jul 21, 2025
01f0f5d
Small corrections to the README.md
piotrczarnas Jul 21, 2025
cff235c
Build a develop version with the most current closed source code.
piotrczarnas Jul 22, 2025
29c0880
Update the version to 1.12.0
piotrczarnas Jul 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
18 changes: 18 additions & 0 deletions .github/workflows/main-publish-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,24 @@ jobs:

- uses: actions/checkout@v3

- name: Configure Git options
run: |
git config pull.rebase false
git config user.email "contact@dqops.com"
git config user.name "DQOps"

- name: Switch to the closed source edition
run: git remote add closedsource https://${{ secrets.PAID_REPOSITORY_USER }}:${{ secrets.PAID_REPOSITORY_TOKEN }}@${{ secrets.PAID_REPOSITORY_HOST }}${{ secrets.PAID_REPOSITORY_PATH }}

- name: Pull the closed source edition
run: git pull -q closedsource closedsource --allow-unrelated-histories --strategy-option=theirs

- name: Checkout the closed source edition
run: git checkout closedsource

- name: Merge changes to the closed source edition
run: git merge ${{ github.ref_name }}

- name: Set up JDK 17
uses: actions/setup-java@v3
with:
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/mvn-build-develop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,26 @@ jobs:

- uses: actions/checkout@v3

- name: Configure Git options
run: |
git config pull.rebase false
git config user.email "contact@dqops.com"
git config user.name "DQOps"

- name: Switch to the closed source edition
run: git remote add closedsource https://${{ secrets.PAID_REPOSITORY_USER }}:${{ secrets.PAID_REPOSITORY_TOKEN }}@${{ secrets.PAID_REPOSITORY_HOST }}${{ secrets.PAID_REPOSITORY_PATH }}

- name: Pull the closed source edition
run: git pull -q closedsource closedsource --allow-unrelated-histories --strategy-option=theirs

- name: Checkout the closed source edition
run: git checkout closedsource

- name: Add SHORT_SHA env property with commit short sha
run: echo "SHORT_SHA=`echo ${GITHUB_SHA} | cut -c1-8`" >> $GITHUB_ENV
run: echo "SHORT_SHA=`echo ``git rev-parse --short=8 HEAD```" >> $GITHUB_ENV

- name: Merge changes to the closed source edition
run: git merge ${{ github.ref_name }}

- name: Set up JDK 17
uses: actions/setup-java@v3
Expand Down
2 changes: 1 addition & 1 deletion .run/dqo run.run.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<option name="region" />
<option name="useCurrentConnection" value="false" />
</extension>
<option name="JAR_PATH" value="$PROJECT_DIR$/dqops/target/dqo-dqops-1.11.6.jar" />
<option name="JAR_PATH" value="$PROJECT_DIR$/dqops/target/dqo-dqops-1.12.0.jar" />
<option name="VM_PARAMETERS" value="-XX:MaxRAMPercentage=60.0 --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED" />
<option name="PROGRAM_PARAMETERS" value="--server.port=8888 --dqo.python.debug-mode=silent" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
Expand Down
9 changes: 6 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# 1.11.6
# 1.12.0

* Shorten the browser caching duration for a list of dashboards.
* Collect statistics for DATETIME types in BigQuery fixed after changes in the BQ interface
* Fixes for working on a machine with a low CPU count (better MaxDOP management for Python processes)
* Current DQ status calculated across different table comparisons.
* Compare the first 1000 records, dropping the rest
* Apply closed source code (2025-07-23)
* Change the license to BUSL 1.1
307 changes: 106 additions & 201 deletions LICENSE

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@ Copyright 2021 and onwards Documati Sp. z o.o.

This product includes software developed at
Documati Sp. z o.o. (https://dqops.com/).

This code branch contains only the code published by Documati Sp. z o.o. for its customers.
It does not contain closed source code owned by Documati Sp. z o.o.
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,33 @@
# DQOps Data Quality Operations Center

DQOps is an DataOps friendly data quality monitoring tool with customizable data quality checks and data quality dashboards.
DQOps comes with around 150 predefined data quality checks which helps you monitor the quality of your data.
DQOps is a DataOps-friendly data quality monitoring tool with customizable data quality checks and data quality dashboards.
DQOps comes with around 150 predefined data quality checks, which help you monitor the quality of your data.

![DQOps screens](https://dqops.com/docs/images/dqo-screens.gif)

## Key features
- Intuitive graphical interface and access via CLI
- Support of a number of different data sources: BigQuery, Snowflake, PostgreSQL, Redshift, SQL Server and MySQL
- ~150 build-in table and column checks with easy customization
- Table and column-level checks which allows writing your own SQL queries
- ~150 built-in table and column checks with easy customization
- Table and column-level checks, which allow writing your own SQL queries
- Daily and monthly date partition testing
- Data grouping by up to 9 different data grouping levels
- Build-in scheduling
- Built-in job scheduling
- Calculation of data quality KPIs which can be displayed on multiple built-in data quality dashboards
- Data quality incident management and notifications

## Installation

To use DQOps you need:
To use DQOps, you need:

- Python version 3.8 or greater (for details see [Python's documentation](https://www.python.org/doc/) and [download sites](https://www.python.org/downloads/)).
- Ability to install Python packages with pip.
- If you want to compile DQOps locally, also Java JDK (version 17 or higher), and a configured JAVA_HOME environment variable.
- If you want to compile DQOps locally, you also need Java JDK (version 17 or higher) and a configured JAVA_HOME environment variable.


DQOps is available on [PyPi repository](https://pypi.org/project/dqops/).

1. To install DQOps via pip manager just run
1. To install DQOps via pip manager, just run

```
python -m pip install --user dqops
Expand Down Expand Up @@ -55,10 +55,10 @@ DQOps is available on [PyPi repository](https://pypi.org/project/dqops/).
After creating a user's home folder, you will be asked whether to log in to the DQO cloud. After typing Y, you will be
redirected to [https://cloud.dqops.com/registration](https://cloud.dqops.com/registration), where you can create a new account, use Google single sign-on (SSO) or log in if you already have an account.

During the first registration, a unique identification code (API Key) will be generated and automatically retrieved by DQOps application.
During the first registration, a unique identification code (API Key) will be generated and automatically retrieved by the DQOps application.
The API Key is now stored in the configuration file.

5. Open the DQOps User Interface Console in your browser by CTRL-clicking on the link displayed on the command line (for example http://localhost:8888)
5. Open the DQOps User Interface Console in your browser by CTRL-clicking on the link displayed on the command line (for example, http://localhost:8888)
or by copying the link.


Expand Down Expand Up @@ -128,7 +128,7 @@ dqops_client = client.Client(base_url="http://localhost:8888")

Alternatively, if you are connecting to a production instance of DQOps that has authentication
enabled, you have to open the user's profile screen in DQOps and generate your DQOps API Key.
Then take the key and use it as the token, when creating an `AuthenticatedClient` instead.
Then take the key and use it as the token when creating an `AuthenticatedClient` instead.

```python
from dqops import client
Expand Down Expand Up @@ -200,7 +200,7 @@ and how to configure data quality checks.

## Contact and issues

If you find any issues with the tool, just post it here:
If you find any issues with the tool, just post them here:

https://github.com/dqops/dqo/issues

Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.11.6
1.12.0
16 changes: 5 additions & 11 deletions distribution/dqo_docker_entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,12 @@
#!/bin/bash
#
# Copyright © 2021 DQOps (support@dqops.com)
# Copyright © 2021-Present DQOps, Documati sp. z o.o. (support@dqops.com)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# This file is licensed under the Business Source License 1.1,
# which can be found in the root directory of this repository.
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Change Date: This file will be licensed under the Apache License, Version 2.0,
# four (4) years from its last modification date.
#

if [ -z "$DQO_JAVA_OPTS" ]; then
Expand Down
10 changes: 5 additions & 5 deletions distribution/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,23 @@

<groupId>com.dqops</groupId>
<artifactId>dqo-distribution</artifactId>
<version>1.11.6</version> <!-- DQOps Version, do not touch (changed automatically) -->
<version>1.12.0</version> <!-- DQOps Version, do not touch (changed automatically) -->
<name>dqo-distribution</name>
<description>DQOps Data Quality Operations Center final assembly</description>
<packaging>pom</packaging>

<properties>
<java.version>11</java.version>
<license.licenseName>apache_v2</license.licenseName>
<license.licenseName>busl_v1_1</license.licenseName>
<compressZip>false</compressZip>
</properties>

<licenses>
<license>
<name>Apache 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<name>BUSL-1.1</name>
<url>https://spdx.org/licenses/BUSL-1.1.html</url>
<distribution>repo</distribution>
<comments>A business-friendly OSS license</comments>
<comments>Business Source License 1.1</comments>
</license>
</licenses>

Expand Down
Loading
Loading