Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON & XML Sources #507

Open
wants to merge 71 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
c11a877
Start implementation (not working yet)
Apr 24, 2024
5202005
Start implementation (not working yet)
Apr 24, 2024
1fa81f6
Fix gradle compile issue
Apr 24, 2024
f294e89
Add json to document conversion
Apr 24, 2024
c248533
Add JSON enumerator
Apr 25, 2024
54746cb
Add namespace and meta functions
Apr 25, 2024
001248f
Fix gradle
Apr 25, 2024
1a4f8af
Add example data
Apr 25, 2024
12aa4e8
Add json conversion
Apr 25, 2024
e56271d
Improve meta retrieval
Apr 30, 2024
f986ca6
Add delegate
May 2, 2024
d1292e8
Start document source refactoring
May 2, 2024
8196df9
Introduce data model handling
May 3, 2024
71977e1
Implement ScanRule, Scan and costs
May 6, 2024
1b88ae4
Fix query execution
Jun 19, 2024
1f9064c
Fix merge conflict
Jun 19, 2024
5ee9966
Fix json queries
Jun 19, 2024
78fdfde
Fix json file path
Jun 19, 2024
f8f01a8
Add directory import
Jun 19, 2024
4de81b6
Add error on modification of collections provided by a source
Jun 27, 2024
794e4d2
Add simple XML source
Jun 27, 2024
01a09c9
Add tests for value conversion
Jun 27, 2024
d0e7cae
Xml enumerator fix
Jul 5, 2024
e53caae
Add error on duplicate files
Jul 17, 2024
0180e22
Add upload and link modes to json source
Jul 17, 2024
a1e3191
Add upload and link modes to xml source
Jul 17, 2024
b96d772
Add useful error message on duplicate file names for JSON source
Jul 17, 2024
d996505
Fix error on dropping document source
Jul 17, 2024
e09e8b7
Fix broken linking on data sources
Jul 19, 2024
32c743e
Add url option to json source
Aug 5, 2024
749c06f
Add url import to xml source
Aug 5, 2024
464fa8b
Add xml file name from url to xml source
Aug 5, 2024
8ae810d
Set useful default value for url
Aug 5, 2024
4ad101b
Fix JSON bugs
Aug 5, 2024
d2c2f43
Fix heap issues on XML source creation
Aug 5, 2024
cb05187
Fix heap issues on XML source enumeration
Aug 5, 2024
4878f08
Merge with master
gartens Aug 6, 2024
445368d
Remove comment
gartens Aug 6, 2024
563356d
Reformat & optimize imports
gartens Aug 6, 2024
bcb7f44
Allow MIT
gartens Aug 6, 2024
25c55fb
Fix XmlToPolyConverterTest
gartens Aug 6, 2024
898960d
Use GenericRuntimeException instead of RuntimeException
gartens Aug 6, 2024
013a474
Add missing dependency on :dbms:shadowJar
gartens Aug 6, 2024
23f98e1
Allow MIT
gartens Aug 6, 2024
b84513c
Add missing dependency on :dbms:shadowJar
gartens Aug 6, 2024
9552b0b
Remove broken upload feature
Aug 7, 2024
84af7ed
Fix leftovers from rebase
gartens Aug 7, 2024
684d340
Fix unit tests
Aug 7, 2024
2606d1a
Fix parser and add more complex example
Aug 8, 2024
c6c1f8a
Various changes according to code review on PR
Aug 11, 2024
4f8d21f
Format
gartens Aug 12, 2024
9501699
Convert ExportedDocument to record class
gartens Aug 19, 2024
ce91406
Remove unused method
gartens Aug 19, 2024
d956739
Convert ExportedColumn into a record
gartens Aug 19, 2024
af8f049
Move implements a class up to AbstractJdbcSource
gartens Aug 19, 2024
a3398de
Use exhaustive switch expressions
gartens Oct 21, 2024
fb7ba52
Use GenericRuntimeException instead of RuntimeException
gartens Oct 21, 2024
80033f2
Use .snapshot instead of .getInstance().getSnapshot()
gartens Oct 21, 2024
d41e57b
Use exhaustive switch expression instead of if
gartens Oct 21, 2024
85f63cc
Fix typos in log messages
gartens Oct 21, 2024
8d833d5
Small improvements
gartens Oct 21, 2024
3ad762e
Remove unused class
gartens Oct 21, 2024
bdb4d90
Remove unused variable
gartens Oct 21, 2024
6ef698c
Annotate parameter as not null
gartens Oct 21, 2024
34eadc0
Replace builder with passing parameters directly
gartens Oct 21, 2024
19fc8f1
Inline simple conversions
gartens Oct 21, 2024
f1e2869
Remove superfluous overrides
gartens Oct 21, 2024
16dce89
Fix typo in JsonToPolyConversion
gartens Oct 21, 2024
e0af943
Annotate parameter as not null
gartens Oct 21, 2024
8d5bc93
Use stricter visibilities for classes & methods
gartens Oct 21, 2024
8a654b2
Remove unused field "fields" from {Json,Xml}Scan
gartens Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use stricter visibilities for classes & methods
gartens committed Oct 21, 2024
commit 8d5bc93b64f23e489089562d1f6d6aeccb739a43
Original file line number Diff line number Diff line change
@@ -34,7 +34,7 @@
import org.polypheny.db.schema.types.TranslatableEntity;
import org.polypheny.db.type.entity.PolyValue;

public class JsonCollection extends PhysicalCollection implements ScannableEntity, TranslatableEntity {
final class JsonCollection extends PhysicalCollection implements ScannableEntity, TranslatableEntity {

private final URL url;
private final Adapter<DocAdapterCatalog> adapter;
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@
import org.polypheny.db.catalog.exceptions.GenericRuntimeException;
import org.polypheny.db.type.entity.PolyValue;

public class JsonEnumerator implements Enumerator<PolyValue[]> {
final class JsonEnumerator implements Enumerator<PolyValue[]> {

private final static ObjectMapper MAPPER = new ObjectMapper();
private final static JsonToPolyConverter CONVERTER = new JsonToPolyConverter();
@@ -22,7 +22,7 @@ public class JsonEnumerator implements Enumerator<PolyValue[]> {
private boolean isCollection;


public JsonEnumerator( URL url ) {
JsonEnumerator( URL url ) {
this.url = url;
}

Original file line number Diff line number Diff line change
@@ -37,9 +37,9 @@
import org.polypheny.db.util.Source;
import org.polypheny.db.util.Sources;

public class JsonMetaRetriever {
final class JsonMetaRetriever {

public static List<ExportedDocument> getDocuments( URL jsonFiles ) throws IOException {
static List<ExportedDocument> getDocuments( URL jsonFiles ) throws IOException {
List<ExportedDocument> exportedDocuments = new LinkedList<>();
Set<String> fileNames = getFileNames( jsonFiles );
ObjectMapper objectMapper = new ObjectMapper();
@@ -62,7 +62,7 @@ public static List<ExportedDocument> getDocuments( URL jsonFiles ) throws IOExce
}


public static URL findDocumentUrl( URL jsonFiles, String name ) throws MalformedURLException, NoSuchFileException {
static URL findDocumentUrl( URL jsonFiles, String name ) throws MalformedURLException, NoSuchFileException {
String[] extensions = { ".json", ".json.gz" };
String path = jsonFiles.getPath();

Original file line number Diff line number Diff line change
@@ -24,9 +24,9 @@
import org.polypheny.db.schema.Namespace;

@Getter
public class JsonNamespace extends Namespace {
final class JsonNamespace extends Namespace {

String name;
private final String name;


public JsonNamespace( String name, long id, long adapterId ) {
Original file line number Diff line number Diff line change
@@ -22,12 +22,12 @@
import org.polypheny.db.plan.AlgOptRuleCall;
import org.polypheny.db.tools.AlgBuilderFactory;

public class JsonProjectScanRule extends AlgOptRule {
final class JsonProjectScanRule extends AlgOptRule {

public static final JsonProjectScanRule INSTANCE = new JsonProjectScanRule( AlgFactories.LOGICAL_BUILDER );
static final JsonProjectScanRule INSTANCE = new JsonProjectScanRule( AlgFactories.LOGICAL_BUILDER );


public JsonProjectScanRule( AlgBuilderFactory algBuilderFactory ) {
private JsonProjectScanRule( AlgBuilderFactory algBuilderFactory ) {
super(
operand( LogicalDocumentScan.class, none() ),
algBuilderFactory,
@@ -40,7 +40,6 @@ public JsonProjectScanRule( AlgBuilderFactory algBuilderFactory ) {
public void onMatch( AlgOptRuleCall call ) {
final LogicalDocumentScan scan = call.alg( 0 );
call.transformTo( new JsonScan( scan.getCluster(), scan.getEntity().unwrapOrThrow( JsonCollection.class ), new int[]{ 0 } ) );

}

}
Original file line number Diff line number Diff line change
@@ -39,14 +39,14 @@
import org.polypheny.db.plan.AlgPlanner;
import org.polypheny.db.plan.AlgTraitSet;

public class JsonScan extends DocumentScan<JsonCollection> implements EnumerableAlg {
public final class JsonScan extends DocumentScan<JsonCollection> implements EnumerableAlg {

@Getter
private final JsonCollection collection;
private final int[] fields;


protected JsonScan( AlgCluster cluster, @NotNull JsonCollection collection, int[] fields ) {
JsonScan( AlgCluster cluster, @NotNull JsonCollection collection, int[] fields ) {
super( cluster, cluster.traitSetOf( EnumerableConvention.INSTANCE ), collection );
this.collection = collection;
this.fields = fields;
Original file line number Diff line number Diff line change
@@ -31,14 +31,14 @@
import org.polypheny.db.type.entity.numerical.PolyLong;
import org.polypheny.db.type.entity.relational.PolyMap;

public class JsonToPolyConverter {
public final class JsonToPolyConverter {

public PolyDocument nodeToPolyDocument( JsonNode node ) {
return new PolyDocument( nodeToPolyMap( node ) );
}


public PolyMap<PolyString, PolyValue> nodeToPolyMap( JsonNode node ) {
private PolyMap<PolyString, PolyValue> nodeToPolyMap( JsonNode node ) {
Map<PolyString, PolyValue> map = new HashMap<>();
node.fields().forEachRemaining( entry -> {
PolyString key = new PolyString( entry.getKey() );
@@ -62,15 +62,15 @@ public PolyValue nodeToPolyValue( JsonNode node ) {
}


public PolyNumber nodeToPolyNumber( JsonNode node ) {
private PolyNumber nodeToPolyNumber( JsonNode node ) {
if ( node.isIntegralNumber() ) {
return new PolyLong( node.asLong() );
}
return new PolyDouble( node.asDouble() );
}


public PolyValue nodeToPolyList( JsonNode node ) {
private PolyValue nodeToPolyList( JsonNode node ) {
return PolyList.of( StreamSupport.stream( node.spliterator(), false )
.map( this::nodeToPolyValue )
.toList() );
Original file line number Diff line number Diff line change
@@ -34,7 +34,7 @@
import org.polypheny.db.schema.types.TranslatableEntity;
import org.polypheny.db.type.entity.PolyValue;

public class XmlCollection extends PhysicalCollection implements ScannableEntity, TranslatableEntity {
final class XmlCollection extends PhysicalCollection implements ScannableEntity, TranslatableEntity {

private final URL url;
private final Adapter<DocAdapterCatalog> adapter;
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
import org.polypheny.db.catalog.exceptions.GenericRuntimeException;
import org.polypheny.db.type.entity.PolyValue;

public class XmlEnumerator implements Enumerator<PolyValue[]> {
final class XmlEnumerator implements Enumerator<PolyValue[]> {

private final static XmlToPolyConverter CONVERTER = new XmlToPolyConverter();

@@ -22,7 +22,7 @@ public class XmlEnumerator implements Enumerator<PolyValue[]> {
private PolyValue[] current;


public XmlEnumerator( URL url ) {
XmlEnumerator( URL url ) {
this.url = url;
initializeReader();
}
Original file line number Diff line number Diff line change
@@ -32,9 +32,9 @@
import org.polypheny.db.util.Source;
import org.polypheny.db.util.Sources;

public class XmlMetaRetriever {
final class XmlMetaRetriever {

public static List<ExportedDocument> getDocuments( URL xmlFiles ) throws IOException {
static List<ExportedDocument> getDocuments( URL xmlFiles ) throws IOException {
List<ExportedDocument> exportedDocuments = new LinkedList<>();
Set<String> fileNames = getFileNames( xmlFiles );

@@ -51,7 +51,7 @@ public static List<ExportedDocument> getDocuments( URL xmlFiles ) throws IOExcep
}


public static URL findDocumentUrl( URL xmlFiles, String name ) throws MalformedURLException, NoSuchFileException {
static URL findDocumentUrl( URL xmlFiles, String name ) throws MalformedURLException, NoSuchFileException {
String[] extensions = { ".xml", ".xml.gz" };
String path = xmlFiles.getPath();

Original file line number Diff line number Diff line change
@@ -24,12 +24,12 @@
import org.polypheny.db.schema.Namespace;

@Getter
public class XmlNamespace extends Namespace {
final class XmlNamespace extends Namespace {

String name;
private final String name;


public XmlNamespace( String name, long id, long adapterId ) {
XmlNamespace( String name, long id, long adapterId ) {
super( id, adapterId );
this.name = name;
}
Original file line number Diff line number Diff line change
@@ -22,12 +22,12 @@
import org.polypheny.db.plan.AlgOptRuleCall;
import org.polypheny.db.tools.AlgBuilderFactory;

public class XmlProjectScanRule extends AlgOptRule {
final class XmlProjectScanRule extends AlgOptRule {

public static final XmlProjectScanRule INSTANCE = new XmlProjectScanRule( AlgFactories.LOGICAL_BUILDER );
static final XmlProjectScanRule INSTANCE = new XmlProjectScanRule( AlgFactories.LOGICAL_BUILDER );


public XmlProjectScanRule( AlgBuilderFactory algBuilderFactory ) {
private XmlProjectScanRule( AlgBuilderFactory algBuilderFactory ) {
super(
operand( LogicalDocumentScan.class, none() ),
algBuilderFactory,
Original file line number Diff line number Diff line change
@@ -39,14 +39,14 @@
import org.polypheny.db.plan.AlgPlanner;
import org.polypheny.db.plan.AlgTraitSet;

public class XmlScan extends DocumentScan<XmlCollection> implements EnumerableAlg {
public final class XmlScan extends DocumentScan<XmlCollection> implements EnumerableAlg {

@Getter
private final XmlCollection collection;
private final int[] fields;


protected XmlScan( AlgCluster cluster, @NotNull XmlCollection collection, int[] fields ) {
XmlScan( AlgCluster cluster, @NotNull XmlCollection collection, int[] fields ) {
super( cluster, cluster.traitSetOf( EnumerableConvention.INSTANCE ), collection );
this.collection = collection;
this.fields = fields;
@@ -83,6 +83,7 @@ public void register( @NotNull AlgPlanner planner ) {
}


@Override
public AlgOptCost computeSelfCost( AlgPlanner planner, AlgMetadataQuery mq ) {
// copied over from the csv project scan rule
return super.computeSelfCost( planner, mq ).multiplyBy( ((double) fields.length + 2D) / ((double) entity.getTupleType().getFieldCount() + 2D) );
Original file line number Diff line number Diff line change
@@ -32,7 +32,7 @@
import org.polypheny.db.type.entity.temporal.PolyTime;
import org.polypheny.db.type.entity.temporal.PolyTimestamp;

public class XmlToPolyConverter {
public final class XmlToPolyConverter {

public PolyDocument toPolyDocument( XMLStreamReader reader, String elementOuterName ) throws XMLStreamException, DecoderException {
return new PolyDocument( toPolyMap( reader, elementOuterName ) );
@@ -62,7 +62,7 @@ private PolyMap<PolyString, PolyValue> toPolyMap( XMLStreamReader reader, String
}


public PolyValue toPolyValue( XMLStreamReader reader ) throws XMLStreamException, DecoderException {
private PolyValue toPolyValue( XMLStreamReader reader ) throws XMLStreamException, DecoderException {
String currentElementName = reader.getLocalName();
String typeName = reader.getAttributeValue( null, "type" );
if ( typeName == null ) {