index.html 8.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167
  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <!DOCTYPE html
  3. PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  4. <html lang="en-us" xml:lang="en-us">
  5. <head>
  6. <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  7. <link rel="stylesheet" type="text/css" href="commonltr.css"/>
  8. <title>Pig Editor</title>
  9. </head>
  10. <body id="xd_583c10bfdbd326ba-3ca24a24-13d80143249--7f9c"><a name="xd_583c10bfdbd326ba-3ca24a24-13d80143249--7f9c"><!-- --></a>
  11. <h1>Sqoop UI</h1>
  12. <p>The Sqoop UI enables transfering data from a relational database
  13. to Hadoop and vice versa. The UI lives uses Apache Sqoop to do this.
  14. See the <a href="http://sqoop.apache.org/docs/1.99.2/index.html">Sqoop Documentation</a> for more details on Sqoop.</p>
  15. <p>
  16. <p>Hue, the <a href="http://gethue.com">open source Big Data UI</a>, has an application that enables transferring data between relational databases and <a href="http://hadoop.apache.org/">Hadoop</a>. This new application is driven by <a href="http://sqoop.apache.org/">Sqoop 2</a> and has several user experience improvements to boot.</p>
  17. <p><iframe frameborder="0" height="495" src="http://player.vimeo.com/video/76063637" width="900"></iframe></p>
  18. <p>Sqoop is a batch data migration tool for transferring data between traditional databases and Hadoop. The first version of Sqoop is a heavy client that drives and oversees data transfer via MapReduce. In Sqoop 2, the majority of the work was moved to a server that a thin client communicates with. Also, any client can communicate with the Sqoop 2 server over its JSON-REST protocol. Sqoop 2 was chosen instead of its predecessors because of its client-server design.</p>
  19. <h2>Importing from MySQL to HDFS</h2>
  20. <p>The following is the canonical import job example sourced from <a href="http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html"><a href="http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html">http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html</a></a>. In Hue, this can be done in 3 easy steps:</p>
  21. <h3>Environment</h3>
  22. <ul><li>
  23. <p>CDH 4.4 or <span>Hue 3.0.0</span></p>
  24. </li>
  25. <li>
  26. <p>MySQL 5.1</p>
  27. </li>
  28. </ul><h3>Troubleshooting</h3>
  29. <p>If the new job button is not appearing, Sqoop2 is probably not starting. Make sure the MySql or other DB connectors are in the /usr/lib/sqoop/lib directory of Sqoop2. Make sure you have these properties in the Sqoop2 Server configuration:</p>
  30. <pre class="code">org.apache.sqoop.repository.schema.immutable=false
  31. org.apache.sqoop.connector.autoupgrade=true
  32. org.apache.sqoop.framework.autoupgrade=true
  33. </pre>
  34. <h3>1. Create a Connection</h3>
  35. <p>In the Sqoop app, the connection manager is available from the “New Job” wizard. To get to the new job wizard, click on “New Job”. There may be a list of connections available if a few have been created before. For the purposes of this demo, we’ll go through the process of creating a new connection. Click “Add a new connection” and fill in the blanks with the data below. Then click save to return to the “New Job” wizard!</p>
  36. <div>
  37. <pre class="code">Connection Parameter Value
  38. Name mysql-connection-demo
  39. JDBC Driver Class com.mysql.jdbc.Driver
  40. JDBC Connection String jdbc:mysql://hue-demo/demo
  41. Username demo
  42. Password demo
  43. </pre>
  44. </div>
  45. <p><br/>Connection form values.<br/><br/></p>
  46. <h3>2. Create a Job</h3>
  47. <p>After creating a connection, follow the wizard and fill in the blanks with the information below.</p>
  48. <div>
  49. <pre class="code">Job Wizard Parameter Value
  50. Name mysql-import-job-demo
  51. Type IMPORT
  52. Connection mysql-connection-demo
  53. Table name test
  54. Storage Type HDFS
  55. Output format TEXT_FILE
  56. Output directory /tmp/mysql-import-job-demo
  57. </pre>
  58. </div>
  59. <p><br/>Job wizard form values.</p>
  60. <h3>3. Save and Submit the Job</h3>
  61. <p>At the end of the Job wizard, click “Save and Run”! The job should automagically start after that and the job dashboard will be displayed. As the job is running, a progress bar below the job listing will be dynamically updated. Links to the HDFS output via the File Browser and Map Reduce logs via Job Browser will be available on the left hand side of the job edit page.</p>
  62. <h1>Sum Up</h1>
  63. <p>The new Sqoop application enables batch data migration from a more traditional databases to Hadoop and vice versa through Hue. Using Hue, a user can move data between storage systems in a distributed fashion with the click of a button.</p>
  64. <p>I’d like to send out a big thank you to the Sqoop community for the new client-server design!</p>
  65. <p>Both projects are undergoing heavy development and are welcoming external contributions! Have any suggestions? Feel free to tell us what you think through <a href="http://groups.google.com/a/cloudera.org/group/hue-user">hue-user</a> or <a href="https://twitter.com/gethue">@gethue</a>​!</p>
  66. </p>
  67. <h2>Installation and Configuration</h2>
  68. <p>The Sqoop UI is one of the applications installed as part of
  69. Hue. For information about installing and configuring Hue, see the Hue Installation
  70. manual.</p>
  71. <h2>Starting</h2>
  72. <p>Click the <strong>Sqoop</strong> icon
  73. (<img alt="image" src="/sqoop/static/art/icon_sqoop_24.png" />) in the navigation bar at the top of
  74. the Hue browser page.</p>
  75. <h2>Sqoop Jobs</h2>
  76. <p>Sqoop UI is oriented around jobs in Apache Sqoop.</p>
  77. <h3>Creating a New Job</h3>
  78. <ol>
  79. <li>Click the <strong>New job</strong> button at the top right.</li>
  80. <li>In the Name field, enter a name.</li>
  81. <li>Choose the type of job: import or export.
  82. The proceeding form fields will change depending on which type is chosen.</li>
  83. <li>Select a connection, or create one if it does not exist.</li>
  84. <li>Fill in the rest of the fields for the job.
  85. For importing, the "Table name", "Storage type", "Output format", and "Output directory" are necessary at a minimum.
  86. For exporting, the "Table name" and "Input directory" are necessary at a minimum.</li>
  87. <li>Click <strong>save</strong> to finish.</li>
  88. </ol>
  89. <h3>Editing a Job</h3>
  90. <ol>
  91. <li>In the list of jobs, click on the name of the job.</li>
  92. <li>Edit the desired form fields in the job.</li>
  93. </ol>
  94. <h3>Copying a Job</h3>
  95. <ol>
  96. <li>In the list of jobs, click on the name of the job.</li>
  97. <li>On the left hand side of the job editor, there should be a panel containing actions.
  98. Click <strong>Copy</strong>.</li>
  99. </ol>
  100. <h3>Removing a Job</h3>
  101. <ol>
  102. <li>In the list of jobs, click on the name of the job.</li>
  103. <li>On the left hand side of the job editor, there should be a panel containing actions.
  104. Click <strong>Delete</strong>.</li>
  105. </ol>
  106. <h3>Running a Job</h3>
  107. <p>There's a status on each of the items in the job list indicating
  108. the last time a job was ran. The progress of the job should dynamically
  109. update. There's a progress bar at the bottom of each item on the job list
  110. as well.</p>
  111. <ol>
  112. <li>In the list of jobs, click on the name of the job.</li>
  113. <li>On the left hand side of the job editor, there should be a panel containing actions.
  114. Click <strong>Run</strong>.</li>
  115. </ol>
  116. <h3>Creating a New Connection</h3>
  117. <ol>
  118. <li>Click the <strong>New job</strong> button at the top right.</li>
  119. <li>At the connection field, click the link titled <strong>Add a new connection</strong>.</li>
  120. <li>Fill in the displayed fields.</li>
  121. <li>Click <strong>save</strong> to finish.</li>
  122. </ol>
  123. <h3>Editing a Connection</h3>
  124. <ol>
  125. <li>Click the <strong>New job</strong> button at the top right.</li>
  126. <li>At the connection field, select the connection by name that should be edited.</li>
  127. <li>Click <strong>Edit</strong>.</li>
  128. <li>Edit the any of the fields.</li>
  129. <li>Click <strong>save</strong> to finish.</li>
  130. </ol>
  131. <h3>Removing a Connection</h3>
  132. <ol>
  133. <li>Click the <strong>New job</strong> button at the top right.</li>
  134. <li>At the connection field, select the connection by name that should be deleted.</li>
  135. <li>Click <strong>Delete</strong>.</li>
  136. </ol>
  137. <p>NOTE: If this does not work, it's like because a job is using that connection.
  138. Make sure not jobs are using the connection that will be deleted.</p>
  139. <h3>Filtering Sqoop Jobs</h3>
  140. <p>The text field in the top, left corner of the Sqoop Jobs page enables fast filtering
  141. of sqoop jobs by name.</p>
  142. </body>
  143. </html>