Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.22 #6

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
7b95636
Updated versiont to 3.22.0
Mar 11, 2021
f2d117e
Revert "SDC-16788. Add override configurations capability to MapR sta…
alejandrosanchezcabana Mar 11, 2021
e34d1b9
Revert "SDC-16789. Add upgrader to Kafka stages for Kafka Additional …
alejandrosanchezcabana Mar 11, 2021
201ffe5
Revert "SDC-16688. Change Kafka Additional Properties validation to m…
alejandrosanchezcabana Mar 11, 2021
1238be1
SDC-16798. Move Connection Selection from PostgresSQL CDC to JDBC tab
zebs97 Mar 12, 2021
37d87e6
SDC-16796 help for the 3.22.0 build
lchen89 Mar 11, 2021
d998212
SDC-16209. SDC picks up installed JDK 11 even when JAVA_HOME points t…
madhukard Mar 15, 2021
c366cb0
Updated version to 3.22.1-SNAPSHOT
Mar 23, 2021
95a84af
SDC-16843. DpmClientInfo from ControlHubResource is NULL
tucu00 Mar 25, 2021
88f8740
Updated version to 3.22.1
Mar 26, 2021
940c5c4
SDC-16806. MQTT Subscriber does not properly restore a persistent ses…
adriaaap Mar 15, 2021
86abc3e
SDC-16847. Oracle CDC raises JDBC_600 'missing redo log files error' …
hprop Mar 28, 2021
028bb07
Updated version to 3.22.2-SNAPSHOT
Apr 1, 2021
b493490
SDC-16784. Treat '' (empty string) as NULL when parsing Log Miner SQL…
lodimas Mar 11, 2021
ffa87f1
SDC-16880. NullPointerException when Oracle CDC origin tries to expir…
lodimas Apr 8, 2021
57d99cc
SDC-16824. Oracle CDC local disk buffering mode does not release disk…
lodimas Apr 8, 2021
5d271b1
SDC-16866. Oracle CDC: LogMiner session can silently fail and provoke…
hprop Apr 2, 2021
5b3bcc0
SDC-16868. Oracle CDC: Truncate timestamp comparisons to seconds prec…
hprop Apr 2, 2021
97b6a4e
SDC-16875. Oracle CDC: Fix edge cases in redo logs selection
hprop Apr 5, 2021
1415299
SDC-16896. Control on duplicate redolog entries for bulk delete on Or…
lodimas Apr 14, 2021
5b2d626
SDC-16876. Oracle CDC: exclude redo logs not belonging to the current…
hprop Apr 9, 2021
8551f69
SDC-16948. Add REDO_VALUE, UNDO_VALUE and Precision Timestamp to the …
lodimas Apr 23, 2021
7cd54e5
SDC-16960. Add sequence of records in Oracle CDC origin
lodimas Apr 26, 2021
3084555
Updated version to 3.22.0
Apr 27, 2021
975dfe2
updated Edge version to 3.22.0
Apr 27, 2021
d8d45b5
Updated version to 3.22.3-SNAPSHOT
May 4, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SDC-16824. Oracle CDC local disk buffering mode does not release disk…
… space until service is restarted.

Upgraded version of MapDB.
Remove transactionallity for file-based maps as we are not relying on it.
Remove all file based maps when Rollback or Expiration.
Improve performance using smaller chunks for incremental file based maps space allocations.
Implement element() method necessary for the expire() method for file based maps.
Force deletion of parent temporary folders when file based maps released and deleted.
Set to null all resources related to file base maps when released to help GC free asap opened mmap file descriptors.
Use specific techniques of the new version of MapDB to enforce freeing FDs and other resources related to file based maps.

Change-Id: Ib7e67f5a536afa21a250b90f5881e1364a19a001
Reviewed-on: https://review.streamsets.net/c/datacollector/+/40620
Tested-by: StreamSets CI <streamsets-ci-spam@streamsets.com>
Reviewed-by: Toni Martinez <toni@streamsets.com>
Reviewed-on: https://review.streamsets.net/c/datacollector/+/40969
Reviewed-by: Hugo Prol <hugo@streamsets.com>
  • Loading branch information
lodimas authored and alejandrosanchezcabana committed Apr 26, 2021
commit 57d99ccc0b506739715b8137802cfd55beb0313c
13 changes: 6 additions & 7 deletions jdbc-protolib/pom.xml
Original file line number Diff line number Diff line change
@@ -48,6 +48,7 @@
<parboiled-java.version>1.1.8</parboiled-java.version>
<postgresql.version>42.2.2</postgresql.version>
<powermock.version>1.7.4</powermock.version>
<mapdb.version>3.0.8</mapdb.version>
</properties>

<dependencies>
@@ -152,6 +153,11 @@
<version>${mssql.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.mapdb</groupId>
<artifactId>mapdb</artifactId>
<version>${mapdb.version}</version>
</dependency>

<!-- Test Dependencies -->
<dependency>
@@ -224,11 +230,6 @@
<version>2.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mapdb</groupId>
<artifactId>mapdb</artifactId>
<version>3.0.5</version>
</dependency>
</dependencies>

<build>
@@ -255,6 +256,4 @@
</plugin>
</plugins>
</build>


</project>
Original file line number Diff line number Diff line change
@@ -15,6 +15,7 @@
*/
package com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle;

import org.apache.commons.io.FileUtils;
import org.jetbrains.annotations.NotNull;
import org.mapdb.DB;
import org.mapdb.DBMaker;
@@ -27,29 +28,42 @@
import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.NoSuchElementException;

/*
We remove all commit-related actions and references as we are using mmap files with no expected failover behavior.
It would be necessary to reconsider this decision if in the future this class relies on other supporting DBMap
entities.
*/
public class FileBackedHashQueue<E> implements HashQueue<E> {

public static final int MB_100 = 100 * 1024 * 1024;
/* Some benchmarks show that there is a huge performance penalty using the old MB_100 (100 * 1024 * 1024) value.
For small transactions, the old size is too big, and transactions are about 50 times slower with it.
For large transactions, there are no essential noticeable differences starting from our proposed size. With
transactions up to 1.000.000 operations, this value seems to be close the its optimum.
*/
public static final int MB_1 = 1024 * 1024;
private HTreeMap underlying;
private final DB db;
private boolean committed = false;
private E tail;
private File folder;

private LinkedHashSet<RsIdSsn> keys = new LinkedHashSet<>();

public FileBackedHashQueue(File file) throws IOException {
this.folder = file;
Files.createDirectories(file.toPath());
Path f = Files.createTempDirectory(file.toPath(), "db-");
db = DBMaker.fileDB(new File(f.toFile(), "dbFile"))
.allocateStartSize(MB_100)
.allocateIncrement(MB_100)
.allocateStartSize(MB_1)
.allocateIncrement(MB_1)
.closeOnJvmShutdown()
.fileDeleteAfterOpen()
.fileDeleteAfterClose()
.fileMmapEnable()
.fileLockDisable()
.transactionEnable()
.cleanerHackEnable()
.fileMmapPreclearDisable()
.make();

underlying = db.hashMap("t").create();
@@ -164,24 +178,40 @@ public E poll() {
@Override
@SuppressWarnings("unchecked")
public E element() {
throw new UnsupportedOperationException();
Iterator<RsIdSsn> it = keys.iterator();
RsIdSsn key = it.next();
if (key == null) {
throw new NoSuchElementException();
}
return (E) underlying.get(key);
}

@Override
public E peek() {
return underlying.isEmpty() ? null : element();
if (!underlying.isEmpty()) {
try {
return element();
} catch (NoSuchElementException eNoSuchElementException) {
return null;
}
}
return null;
}

public void close() {
this.underlying.close();
this.db.close();
try {
FileUtils.deleteDirectory(folder);
underlying = null;
keys = null;
} catch (Throwable eThrowable) {
}
}

@Override
public void completeInserts() {
if (!committed) {
db.commit();
committed = true;
}
return;
}

private class FileBackedHashQueueIterator implements Iterator<E> {
Original file line number Diff line number Diff line change
@@ -878,6 +878,9 @@ private int processBufferedTransaction(TransactionIdKey key, LogMinerRecord reco
if (record.getOperationCode() == ROLLBACK_CODE || record.getScn().compareTo(lastCommitSCN) < 0) {
bufferedRecordsLock.lock();
try {
HashQueue<RecordSequence> records = bufferedRecords.getOrDefault(key, EMPTY_LINKED_HASHSET);
records.completeInserts();
records.close();
bufferedRecords.remove(key);
LOG.debug(ROLLBACK_MESSAGE, key.txnId);
} finally {
@@ -2000,6 +2003,11 @@ private void discardOldUncommitted(LocalDateTime startTime) {
}
}
txnDiscarded.incrementAndGet();
HashQueue<RecordSequence> records = entry.getValue();
if (records != null) {
records.completeInserts();
records.close();
}
iter.remove();
}
}
Original file line number Diff line number Diff line change
@@ -74,8 +74,8 @@ public void testBuildTableConditionSchemaPatterns() {
LogMinerSession.Builder builder = new LogMinerSession.Builder(Mockito.mock(Connection.class), 19);
String condition = builder.buildTablesCondition(tables);
Assert.assertEquals(
"((SEG_OWNER LIKE '_SYS_' AND (TABLE_NAME LIKE '%PATTERN1%' OR TABLE_NAME LIKE '%PATTERN2%'))" +
" OR (SEG_OWNER LIKE '%SDC%' AND (TABLE_NAME IN ('TABLE1'))))",
"((SEG_OWNER LIKE '%SDC%' AND (TABLE_NAME IN ('TABLE1')))" +
" OR (SEG_OWNER LIKE '_SYS_' AND (TABLE_NAME LIKE '%PATTERN1%' OR TABLE_NAME LIKE '%PATTERN2%')))",
condition
);