Loading JSON
Moving data into data services should be easy. We can help.
On the IBM Cloud, it should be easy to move data in, share it between services, and get data out. These projects represent our ongoing work to make data import and export easier for users.
Importing structured data into document data stores
Structured data living in spreadsheets, relational databases, or as flat CSV/TSV files can be easily imported into IBM Cloudant, MongoDB or ElasticSearch using our nosqlimport tool.
Cloudant and CouchDB have their own tool, the popular couchimport utility which allows structured data to be bulk imported into the JSON document stores, with an optional transformation along the way. The couchimport utility is also packaged as an installable Electron app to make data transfer as simple as a “drag and drop.”
Moving data from other clouds
Developers are keen to avoid being locked in to a cloud platform so we have built tools to extract data from other cloud databases, which can easily be paired with couchimport or nosqlimport to move the data elsewhere.
Backing up Cloudant
Cloudant is an operational database, safely storing vital data for our customers in distributed clusters. Customer backup needs vary widely, but can be scripted easily with the couchbackup utility which can turn databases into text files, to allow rotating backups to be archived on cheaper object storage.
Transporter
The nice folks at Compose have written the Transporter utility that makes data movement between files and their MongoDB, ElasticSearch, RethinkDB, PostgreSQL, and RabbitMQ products.
Projects
-
Elasticsearch, Bulk Uploading and the High-Level Java REST Client - Part 2
GitHub | Compose ArticlesHow to use the BulkProcessor to bulk up all your Elasticsearch updates.
-
Elasticsearch, Bulk Uploading and the High-Level Java REST Client - Part 1
GitHub | Compose ArticlesA practical look at the relatively new High-Level Java REST Client for Elasticsearch uploads.
-
Cloudant and CouchDB Replication with couchreplicate
GitHub | MediumA new command-line tool for managing and monitoring multiple replication jobs.
-
Tuning it up to 11 - Transporter and the Cloudant Adaptor
GitHub | Compose ArticlesRevisiting the simple Cloudant adaptor to make it a bit smarter.
-
Keeping Local Documents Local in a CouchDB-Style Replication Job
MediumFor when you replicate data, but need to leave a bit of config behind.
-
Building the Soul of a New Transporter Adaptor
Compose ArticlesTaking the first steps to creating our own Transporter adapter for Cloudant.
-
Building Your Own Data Transport with Compose Transporter
Compose ArticlesA stodge-free open source ETL tool you can extend in Go. Here's how.
-
Using Your Own Domain Name With Cloudant
GitHub | MediumSecure your custom domain using serverless functions on the IBM Cloud.
-
Building a Serverless Data Pipeline
GitHub | Medium | YouTubeData transformation is a great use case for serverless technology.
-
Search in Reverse: Real-Time Updates for Data as It Enters Your System
Compose ArticlesDiscover space on twitter using Elasticsearch's percolate query.
-
Couchimport: Import flat data into CouchDB efficiently.
npm | GitHub | MediumLatest update adds one-to-many transformations.
-
Visualizing Data with Jupyter Notebooks, PixieDust, and Compose MongoDB
Compose ArticlesConnecting to Compose MongoDB and creating rich presentations for your data inside a Jupyter notebook is made easier with PixieDust.
-
The Transporter 0.4 series has begun
GitHub | Compose ArticlesCompose's Transporter is a powerful way of transporting data between different databases. With each iteration, we look to improve it.
-
Custom Cloudant Replication
GitHub | MediumEmulating filtered replication using the changes feed and OpenWhisk functions.
-
Diff Your Couch-Style Databases with couchdiff
GitHub | Medium | npmCommand-Line tool for comparing two Apache CouchDB™ or Cloudant databases.
-
Moving data from DynamoDB to Cloudant or CouchDB
GitHub | Medium | npmIntroducing the dynamodbexport npm module.
-
Installing Web Apps with Electron
GitHub | MediumA simple CouchDB data importer app that also shows how to bundle a web app for native distribution with Electron.
-
Seven Databases in Seven Days
GitHub | developerWorksA quick tour of (some of) our database services.
-
CouchDB Document Copy and Transform Service
GitHub | developerWorksAdd some T to your ETL for CouchDB.
-
Importing JSON Documents with nosqlimport
GitHub | developerWorks | npmMoving data doesn't have to stink.
-
Simple CouchDB and Cloudant Backup
GitHub | developerWorks | npmBack up your data, without breaking your back.