<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">

<document>
  <header>
    <title>Reader and Writer Interfaces</title>
  </header>
  <body>
  
<p>HCatalog provides a data transfer API for parallel input and output
without using MapReduce. This API provides a way to read data from a
Hadoop cluster or write data into a Hadoop cluster, using a basic storage
abstraction of tables and rows.</p>
<p>The data transfer API has three essential classes:</p>
<ul>
    <li>HCatReader &ndash; reads data from a Hadoop cluster</li>
    <li>HCatWriter &ndash; writes data into a Hadoop cluster</li>
    <li>DataTransferFactory &ndash; generates reader and writer instances</li>
</ul>
<p>Auxiliary classes in the data transfer API include:</p>
<ul>
    <li>ReadEntity</li>
    <li>ReaderContext</li>
    <li>WriteEntity</li>
    <li>WriterContext</li>
</ul>
<p>The HCatalog data transfer API is designed to facilitate integration
of external systems with Hadoop.</p>

<!-- ==================================================================== -->
<section>
<title>HCatReader</title>

<p>Reading is a two-step process in which the first step occurs on
the master node of an external system. The second step is done in
parallel on multiple slave nodes.</p>

<p>Reads are done on a “ReadEntity”. Before you start to read, you need to define
a ReadEntity from which to read. This can be done through ReadEntity.Builder. You
can specify a database name, table name, partition, and filter string.
For example:</p>

<source>
ReadEntity.Builder builder = new ReadEntity.Builder();
ReadEntity entity = builder.withDatabase(“mydb”).withTable("mytbl").build();
</source>

<p>The code snippet above defines a ReadEntity object ("<code>entity</code>"),
comprising a table named “mytbl” in a database named “mydb”, which can be used
to read all the rows of this table.
Note that this table must exist in HCatalog prior to the start of this
operation.</p>

<p>After defining a ReadEntity, you obtain an instance of HCatReader
using the ReadEntity and cluster configuration:</p>

<source>
HCatReader reader = DataTransferFactory.getHCatReader(entity, config);
</source>

<p>The next step is to obtain a ReaderContext from <code>reader</code>
as follows:</p>

<source>
ReaderContext cntxt = reader.prepareRead();
</source>

<p>All of the above steps occur on the master node. The master node then
serializes this ReaderContext object and sends it to all the slave nodes.
Slave nodes then use this reader context to read data.</p>

<source>
for(InputSplit split : readCntxt.getSplits()){
HCatReader reader = DataTransferFactory.getHCatReader(split,
readerCntxt.getConf());
       Iterator&lt;HCatRecord&gt; itr = reader.read();
       while(itr.hasNext()){
              HCatRecord read = itr.next();
          }
}
</source>

</section>

<!-- ==================================================================== -->
<section>
<title>HCatWriter</title>

<p>Similar to reading, writing is also a two-step process in which the first
step occurs on the master node. Subsequently, the second step occurs in
parallel on slave nodes.</p>

<p>Writes are done on a “WriteEntity” which can be constructed in a fashion
similar to reads:</p>

<source>
WriteEntity.Builder builder = new WriteEntity.Builder();
WriteEntity entity = builder.withDatabase(“mydb”).withTable("mytbl").build();
</source>

<p>The code above creates a WriteEntity object ("entity") which can be used
to write into a table named “mytbl” in the database “mydb”.</p>

<p>After creating a WriteEntity, the next step is to obtain a WriterContext:</p>

<source>
HCatWriter writer = DataTransferFactory.getHCatWriter(entity, config);
WriterContext info = writer.prepareWrite();
</source>

<p>All of the above steps occur on the master node. The master node then
serializes the WriterContext object and makes it available to all the
slaves.</p>

<p>On slave nodes, you need to obtain an HCatWriter using WriterContext
as follows:</p>

<source>
HCatWriter writer = DataTransferFactory.getHCatWriter(context);
</source>

<p>Then, <code>writer</code> takes an iterator as the argument for
the <code>write</code> method:</p>

<source>
writer.write(hCatRecordItr);
</source>

<p>The <code>writer</code> then calls getNext() on this iterator in a loop
and writes out all the records attached to the iterator.</p>
</section>

<!-- ==================================================================== -->
<section>
    <title>Complete Example Program</title>
<p>A complete java program for the reader and writer examples above can be found at: <a href="https://svn.apache.org/repos/asf/hive/trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestReaderWriter.java">https://svn.apache.org/repos/asf/hive/trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestReaderWriter.java</a>.</p>

</section>


    
  </body>
</document>
