2021-04-05 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, April 05, 2021 - 9:30am EST / 2:30pm BST

// First Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
James McLeod	FINOS	@mcleo-d
Alvin Shih	Morgan Stanley	@alvin-c-shih
Aaron Davies	Morgan Stanley	@adavies42
Peter Gyorok	Morgan Stanley	@gyorokpeter
Alok Dutt	Citi

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

• (5 min) Convene, roll call, welcome new people.
  • Display FINOS Antitrust Policy summary slide
  • Review Meeting Notices (see above)
• (5 min) Approve previous meeting minutes.
  • It was pointed out that the WebEx link on the Wiki needs updating.
    • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• (5 min) Wrapper of psutil (Python) in Appendix B of Minutes for 2021-03-01.
  • Please test and see if you'd like to turn it into a full-fledged module for FINOS.
• (15 min) Shall we hard fork qPython? https://github.com/exxeleron/qPython
  • Only supports protocol 3 whereas we have users who want >2 GB payloads, which implies at least protocol 5.
    • https://code.kx.com/q/basics/ipc/
  • No appetite from the qPython maintainers.
    • IPC 3.4 Support/Upgrade exxeleron/qPython#58
  • qPython and c.java read the entire payload into a buffer before decompressing.
    • https://github.com/exxeleron/qPython/blob/master/qpython/fastutils.pyx
    • Significant rework to decompress on the fly.
• (10 min) Synthetic data for microbenchmarks?
  • SciDB makes use of sample TAQ data.
    • https://github.com/Paradigm4/TAQ
    • See: https://www.nyse.com/market-data/historical/daily-taq
    • Is there a particular subset of interest?
• (5 min) AOB, Q&A, Adjourn.
  • Ideas for future calls
    • entitlements
    • tools for setting up multiple cooperating q processes
    • spreadsheet / MarkDown file comparing compression schemes

Decisions Made

• psutil / Unique Set Size
  • Attendees expressed preference for implementation that doesn't depend on Python.
• qPython hard fork
  • Agreement to move forward.
  • Suggested name for fork is qPython64.
  • Aim for most tactical fix to get >2 GB payloads.
  • Use cases that would benefit include:
    • gateways that combine RDB+HDB results
    • heavily traded names such as QQQ
• benchmarking
  • MIght want to look at TPC-DS.
    • http://www.tpc.org/information/benchmarks5.asp
  • STAC is aware of our interest and will discuss with members.
  • Kdb's Data At Rest Encryption activated in a manner similar to on-disk compression
    • https://code.kx.com/q/kb/dare/

-36!(`:testkey.key;"mypassword")
(`:etest;20;16;0)set 100000000?10000
system"ts max get`:etest"

WebEx info

Webex

• https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

• US: +1-415-655-0003 US Toll
• UK: +44-20319-88141 UK Toll
• Access code: 663 746 304
• GitHub Repos:
  • https://github.com/finos/kdb
  • https://github.com/finos/kdb-studio
• Wiki Page:
  • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• KDB Metrics on FINOS
  • https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

[adavies42]

hi

[alvin-c-shih]

Howdy!

[gyorokpeter]

hi