AVRO & ORC File-format Implementation In IBM Information Server 11.5

Vik Malhotra, PMP

(graphs) - [:ARE] - > (everywhere)

Published Jan 21, 2016

+ Follow

AVRO & ORC File-format Implementation in File Connector Stage In Information Server 11.5

In my earlier posts on File connector stage in DataStage (Information Server 11.5), I discussed how it can be leveraged to read files from and write files to a local file system on the engine tier or a Hadoop Distributed File System (HDFS) by using the WebHDFS API or the HttpFS API.

The File connector supports HDFS by using the WebHDFS API and the HttpFS API and is hence independent of the HDFS distribution and version.

The File connector supports Kerberos authentication.
The File connector supports creating Hive tables.

In this post I show how to leverage the new file format features added to File Connector namely Avro (Deflate, Snappy & Bzip2 compression supported) and ORC (zlib, snappy & lzo compression supported) formats.

All the above and new capabilities require “zero coding” and are configured on a graphical interface in DataStage designer.

The detailed document is available on request, please message me over linkedin.

Download link for AVRO & ORC File-format Implementation In IBM Information Server 11.5 > http://bit.ly/1JZWfVi

Sajjad Shafi

would you please share the detail document in regars to avro and orc

Kleber Rebello Jr

Vik Malhotra Thx.

the link is in the post above - Download link for AVRO & ORC File-format Implementation In IBM Information Server 11.5 > http://bit.ly/1JZWfVi

Yogi Golle

Hi Vic, thanks for this post. I am trying to write a sequential file on local Datastage server to be landed as an Avro file in Hadoop. I have a local schema file for the sequential file. Is there an easy way convert that schema file to an Avro schema for the Avro file connection?

Victoria Suchoverchova

Vik Malhotra Thanks Vic, I will try!

Victoria Suchoverchova I used the new hive connector to write a simple avro hive table, it does not require schema file. If you need a schema file for the file connector then, write a avro table with few records with the Hive connector and then use the command below to generate the schema file. This schema file can be used in File connector. [cloudera@quickstart ~]$ java -jar /usr/lib/avro/avro-tools.jar getschema hdfs://quickstart.cloudera:8020/user/dsadm/data_dir/geolocation_avro/000000_0 > geolocation.avsc

Hi Vik, thank you for the nice documentation, it's really helpful. Do you have maybe any inputs how to create Avro file via File Connector with struct Hadoop data type?

Meena kannan

Hi Vik, Am not able to access or download the following document. Could you please share me the document on how to read/write ORC file using file connector stage

Atul Rao

Good article Vik. Does the connector handle custom serde jars if it is part of my hive create table process?

See more comments

AVRO & ORC File-format Implementation in File Connector Stage In Information Server 11.5

Kafka connector DataStage Jobs in Information Server 11.5

Apr 12, 2017

Hive Table Partitioning DataStage Jobs With IIS Hive Connector (Part 2)

Apr 1, 2017

Hive File Formats and Compression DataStage Jobs With Information Server Hive Connector (Part 1)

Mar 30, 2017