node-pg-copy-streams - COPY FROM / COPY TO for node-postgres

  •        347

COPY FROM / COPY TO for node-postgres. Stream from one database to another, and stuff. PostgreSQL supports streaming binary data directly into and out of a table. This means you can take your favorite CSV or TSV or whatever format file and pipe it directly into an existing PostgreSQL table.

https://github.com/brianc/node-pg-copy-streams

Tags
Implementation
License
Platform

   




Related Projects

rdb - Javascript ORM

  •    Javascript

ORM for nodejs. Supports postgres, mySql and sqlite. 1.7.1 Support for schemas (postgres only). 1.7.0 sqlite3 is now a peer dependency. Add it to your own package.json if you intend to use it. 1.6.9 Bugfix: one-to-many relation returns empty if strategy is included. 1.6.8 Bugfix: one-to-many relation returns empty if insert/update is done earlier in transaction. 1.6.7 Bugfix in relations. 1.6.6 Bugfix. 1.6.5 Improved performance on relations. 1.6.4 Bugfix. 1.6.3 Bugfix: potential incorrect timeZoneOffset when serializing date to JSON. Got timeZoneOffset from now() instead of on actual date. 1.6.2 Removed es6 syntax to ensure backwards compatability. Fixed global var leak. 1.6.1 Now supporting sqlite. 1.6.0 Bugfix: potential ambigous column error when using limit and relating to other tables. 1.5.9 Bugfix: using multipleStatements in mySql could sometimes cause an error when updates are run right before a select. Improved performance on limit when relating to other tables. Using uuid instead of node-uuid Updated all dependencies but generic-pool to latest. (Generic-pool has some breaking changes in latest. I will update it in next release.) 1.5.8 Cleanup line breaks in documentation. 1.5.7 Bugfix: getById.exclusive and tryGetById.exclusive did not lock if row was cached. Improved performance on tryGetFirst. 1.5.6 Raw sql filters can accept sql both as string and as function. E.g. var filter = {sql: function() {return 'foo > 1';}}. 1.5.5 Optional locks for getMany, tryGetFirst and tryGetById. Instead of calling getMany(params) just call getMany.exclusive(params). Same syntax goes for tryGetFirst and tryGetById. This will result in SELECT FOR UPDATE. Bugfix: bulk deletes now accepts raw sql filters too. 1.5.4 Transaction locks. Postgres only. 1.5.3 Upgraded to pg 6.0.3 1.5.2 Improved performance and reduced memory footprint. 1.5.1 Documented JSON column type. Bug fix: Insert and foreign key violation. 1.5.0 JSON column type. Postgres json type does not support rdb filters. 1.4.1 Empty filter would sometimes cause invalid filter. 1.4.0 Raw SQL query. 1.3.0 getMany() now supports limit and orderBy - same syntax as in streaming. 1.2.3 Bugfix: iEqual gave incorrect sql when parameterized. 1.2.2 Exlusive no longer returns a clone of table. It has changes current table to exclusive locking. 1.2.1 Bugfix: Exclusive row locks 1.2.0 Exclusive row locks 1.1.0 Now supporting streaming. Requires postgres or MySQL >=5.7.7 1.0.8 README fixup. 1.0.7 Better performance on insert and update. 1.0.6 Bugfix: Transaction domain should not forward rdb singleton from old domain. 1.0.5 Documentation cleanup. 1.0.4 orderBy in toDto(). 1.0.3 toDto() using next tick on every thousandth row to avoid maximum call stack size exceeded. 1.0.2 Reduced number of simultaneous promises in order to avoid maximum call stack size exceeded. 1.0.1 Bugfix: Incorrect insert/updates on timestamp without timezone. The time was converted utc instead of stripping the timezone. 1.0.0 Transaction domain forwards properties from old domain. Semantic versioning from now on. 0.5.1 Improved performance 0.5.0 Logging: rdb.log(someFunc) logs sql and parameters. Raw sql filters. 0.4.9 New method: tryGetById. New filter: iEqual, postgres only. Bugfix: rows.toJSON() without strategy did not include any children. 0.4.8 Explicit pooling with size and end(). Bugfix: mySql did not release client to pool. 0.4.7 Upgraded to pg 4.3.0 Upgraded to mysql 2.5.5 0.4.6 Upgraded pg 4.2.0. 0.4.5 Oops. Forgot to use pg.js instead of pg. 0.4.4 Upgraded all dependencies to latest. Using pg.js instead of pg. 0.4.3 Can ignore columns when serializing to dto. 0.4.2 Bugfix: update on a row crashes when a delete occurs earlier in same transaction. 0.4.1 Bugfix: more global leaks. 0.4.0 Bugfix: global leak. 0.3.9 Bugfix: eager loading joins/hasOne with non unique column names was not handled correctly. 0.3.8 Supports mySql. Bulk deletes. 0.3.7 Bugfix: eager loading manyRelation on a join/hasOne returned empty array #11 0.3.6 Fixed sql injection vulnerability. 0.3.5 Built-in fetching strategies for lazy loading. Works best in readonly scenarios. 0.3.4 Docs and examples split moved to separate file. 0.3.3 Fixed documentation layout again. 0.3.2 Fixed documentation layout. 0.3.1 Case insensitive filters: iStartsWith, iEndsWith and iContains. 0.3.0 Fix broken links in docs. 0.2.9 Support for row.delete(). Rollback only throws when error is present. 0.2.8 Guid accepts uppercase letters. Bugfix: null inserts on guid columns yielded wrong sql. 0.2.7 New method, toDto(), converts row to data transfer object. Bugfix: toJSON returned incorrect string on hasMany relations. 0.2.6 Fixed incorrect links in README. 0.2.5 Bugfix: caching on composite keys could give a crash #7. Improved sql compression on insert/update. 0.2.4 Bugfix: getMany with many-strategy and shallowFilter yields incorrect query #6. 0.2.3 Reformatted documentation. No code changes.

pqstream - pqstream turns your postgres database into an event stream

  •    Go

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

hstream - The database built for IoT streaming data storage and real-time stream processing.

  •    Haskell

The database built for IoT streaming data storage and real-time stream processing. By subscribing to streams in HStreamDB, any update of the data stream will be pushed to your apps in real-time, and this promotes your apps to be more responsive.

stream-handbook - how to write node programs with streams

  •    Javascript

Now you will have a stream-handbook command that will open this readme file in your $PAGER. Otherwise, you may continue reading this document as you are presently doing.Streams come to us from the earliest days of unix and have proven themselves over the decades as a dependable way to compose large systems out of small components that do one thing well. In unix, streams are implemented by the shell with | pipes. In node, the built-in stream module is used by the core libraries and can also be used by user-space modules. Similar to unix, the node stream module's primary composition operator is called .pipe() and you get a backpressure mechanism for free to throttle writes for slow consumers.


node-pg-migrate - Node.js database migration management for Postgresql

  •    Javascript

Node.js database migration management built exclusively for postgres. (But can also be used for other DBs conforming to SQL standard - e.g. CockroachDB.) Started by Theo Ephraim, now maintained by Salsita Software. You can specify your database connection information using config.

pq - Pure Go Postgres driver for database/sql

  •    Go

pq is a pure Go postgres driver for Go's database/sql package.

node-postgres - PostgreSQL client for node.js

  •    Javascript

Non-blocking PostgreSQL client for Node.js. Pure JavaScript and optional native libpq bindings. It supports Connection pooling, PostgreSQL data-type coercion, Parameterized queries, Named statements with query plan caching, Async notifications with LISTEN/NOTIFY and lot more.

Stream-Framework - Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis

  •    Python

Stream Framework is a python library which allows you to build activity streams & newsfeeds using Cassandra and/or Redis. If you're not using python have a look at Stream, which supports Node, Ruby, PHP, Python, Go, Scala, Java and REST. Stream Framework's authors also offer a web service for building scalable newsfeeds & activity streams at Stream. It allows you to create your feeds by talking to a beautiful and easy to use REST API. There are clients available for Node, Ruby, PHP, Python, Go, Scala and Java. The Get Started page explains the API & concept in a few clicks. It's a lot easier to use, free up to 3 million feed updates and saves you the hassle of maintaining Cassandra, Redis, Faye, RabbitMQ and Celery workers.

OpenTheatre - Search movies, series, anime, subtitles, torrents from open directories

  •    CSharp

OpenTheatre allows you to search for movies, series, anime, subtitles and torrents files available hosted all over the Internet found in open directories. We've gathered all the appropriate files out there so you don't have to. Created by a few friends who wanted to come up with something different.

alpakka - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka

  •    Scala

Systems don't come alone. In the modern world of microservices and cloud deployment, new components must interact with legacy systems, making integration an important key to success. Reactive Streams give us a technology-independent tool to let these heterogeneous systems communicate without overwhelming each other. The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure. Akka Streams is a Reactive Streams and JDK 9+ java.util.concurrent.Flow-compliant implementation and therefore fully interoperable with other implementations.

postgres-copy - Simple PostgreSQL's COPY command support in ActiveRecord models

  •    Ruby

This Gem will enable your AR models to use the PostgreSQL COPY command to import/export data in CSV format. If you need to tranfer data between a PostgreSQL database and CSV files, the PostgreSQL native CSV parser will give you a greater performance than using the ruby CSV+INSERT commands. I have not found time to make accurate benchmarks, but in the use scenario where I have developed the gem I have had a four-fold performance gain. This gem was written having the Rails framework in mind, I think it could work only with active-record, but I will assume in this README that you are using Rails.

rtsp-simple-server - ready-to-use server and proxy that allows to read, publish and proxy video and audio streams with RTSP, RTMP, HLS

  •    Go

rtsp-simple-server is a ready-to-use and zero-dependency server and proxy that allows users to publish, read and proxy live video and audio streams through various protocols like RTSP, RTMP, HLS. It publish and read live streams to the server. It acts as a proxy and serve streams from other servers or cameras, always or on-demand.

node-copy-paste - A command line utility that allows read/write (i

  •    Javascript

A command line utility that allows read/write (i.e copy/paste) access to the system clipboard. It does this by wrapping pbcopy/pbpaste (for OSX), xclip (for Linux, FreeBSD, and OpenBSD), and clip (for Windows). Currently works with node.js v0.8+. copy(text[, callback]): asynchronously replaces the current contents of the clip board with text. Takes either a string, array, object, or readable stream. Returns the same value passed in. Optional callback will fire when the copy operation is complete.

message-db - Microservice native message and event store for Postgres

  •    Shell

A fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Evented Microservices applications. An event sourcing and Pub/Sub message store built on Postgres for simple cloud or local hosting. A minimalist implementation of the essential features of tools like Event Store or Kafka, with built-in support for messaging patterns like Pub/Sub, and consumer patterns like consumer groups.

forceps - Copy active record models from remote databases

  •    Ruby

Forceps lets you copy related models from one database into another. The source and target databases must support an active record connection. Typically, your source database is a remote production database and your target database is a local development one. The low level connection mechanism doesn't matter thanks to Active Record. For example, this gem has been tested on MySQL tunneled over SSH and on Heroku using Postgres credentials.

websocket-stream - websockets with the node stream API

  •    Javascript

Use HTML5 websockets using the Node Streams API.This module works in Node or in Browsers that support WebSockets. You can use browserify to package this module for browser use.

knex.js - A SQL query builder that is flexible, portable, and fun to use

  •    Javascript

Knex.js is a "batteries included" SQL query builder for Postgres, MSSQL, MySQL, MariaDB, SQLite3, Oracle, and Amazon Redshift designed to be flexible, portable, and fun to use. It features both traditional node style callbacks as well as a promise interface for cleaner async flow control, a stream interface, full featured query and schema builders, transaction support (with savepoints), connection pooling and standardized responses between different query clients and dialects.

winton-kafka-streams - A Python implementation of Apache Kafka Streams

  •    Python

Implementation of Apache Kafka's Streams API in Python. Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. Kafka has Streams API added for building stream processing applications using Apache Kafka. Applications built with Kafka's Streams API do not require any setup beyond the provision of a Kafka cluster.

streamDM - Stream Data Mining Library for Spark Streaming

  •    Scala

streamDM is a new open source software for mining big data streams using Spark Streaming, started at Huawei Noah's Ark Lab. streamDM is licensed under Apache Software License v2.0. Big Data stream learning is more challenging than batch or offline learning, since the data may not keep the same distribution over the lifetime of the stream. Moreover, each example coming in a stream can only be processed once, or they need to be summarized with a small memory footprint, and the learning algorithms must be very efficient.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.