Apache Hadoop

https://hadoop.apache.org/docs/stable
Admin
Apache Hadoop is a framework that enables distributed processing of large data sets across clusters...
Tokens:2,567,315
Snippets:28,482
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Get Setup Progress

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/JobStatus.html

Returns the percentage of progress in setup.

```Java
public float setupProgress()
```

--------------------------------

### Start a Service

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html

This example shows how to start a service that has been previously stopped or created in a stopped state. Starting a service will transition all its components to a running state.

```json
{
  "state": "RUNNING"
}
```

--------------------------------

### setup

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.html

Called once at the start of the task. This method is intended for initialization.

```APIDOC
## setup

### Description
Called once at the start of the task.

### Method
```java
protected void setup(org.apache.hadoop.mapreduce.Reducer.Context context)
```

### Overrides
`setup` in class `Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>`
```

--------------------------------

### Start Dynamometer Cluster (Manual)

Source: https://hadoop.apache.org/docs/stable/hadoop-dynamometer/Dynamometer.html

Launches the infrastructure application to start the internal HDFS cluster. Demonstrates required arguments for cluster setup.

```bash
./dynamometer-infra/bin/start-dynamometer-cluster.sh
    -hadoop_binary_path hadoop-3.0.2.tar.gz
    -conf_path my-hadoop-conf
    -fs_image_dir hdfs:///fsimage
    -block_list_path hdfs:///dyno/blocks

```

--------------------------------

### Example GET Request for Listing Domains

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

An example of how to make a GET request to list domains for a specific owner, 'alice'.

```http
GET http://localhost:8188/ws/v1/timeline/domain?owner=alice

```

--------------------------------

### start()

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/service/AbstractService.html

Starts the service.

```APIDOC
## start()

### Description
Start the service.
```

--------------------------------

### setupProgress

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/Job.html

Gets the progress of the job's setup tasks as a float between 0.0 and 1.0.

```APIDOC
## setupProgress()

### Description
Get the _progress_ of the job's setup-tasks, as a float between 0.0 and 1.0.

### Method
`float`

### Endpoint
N/A (SDK method)
```

--------------------------------

### Running Java Client Example

Source: https://hadoop.apache.org/docs/stable/hadoop-auth/Examples.html

Instructions to run the Java client example, including navigating to the examples directory and using Maven to execute the Java code. Assumes Kerberos ticket initialization.

```bash
$ kinit
Please enter the password for tucu@LOCALHOST:

$ cd examples

$ mvn exec:java -Durl=http://localhost:8080/hadoop-auth-examples/kerberos/who
```

--------------------------------

### Get Adjusted Start Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/io/compress/SplitCompressionInputStream.html

Retrieves the adjusted start offset after the codec might have modified it. Use this to get the effective start of the data range.

```java
public
long
getAdjustedStart()
```

--------------------------------

### NMClientAsync Usage Example

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/client/api/async/NMClientAsync.html

Demonstrates the lifecycle management and common operations of NMClientAsync, including creating the client, initializing, starting, managing containers, and stopping the client.

```APIDOC
## NMClientAsync Lifecycle and Operations

### Description
This example illustrates how to use `NMClientAsync` to manage NodeManager interactions asynchronously. It covers the typical workflow from client creation to stopping the client, including starting and querying containers.

### Usage
```java
// Create an instance of NMClientAsync with a custom CallbackHandler
NMClientAsync asyncClient = NMClientAsync.createNMClientAsync(new MyCallbackHandler());

// Initialize the client with configuration
asyncClient.init(conf);

// Start the client
asyncClient.start();

// Start a container
asyncClient.startContainer(container, containerLaunchContext);

// ... wait for container to be started ...

// Get the status of a container
asyncClient.getContainerStatus(container.getId(), container.getNodeId(), container.getContainerToken());

// ... handle the status in the callback instance ...

// Stop a container
asyncClient.stopContainer(container.getId(), container.getNodeId(), container.getContainerToken());

// ... wait for container to be stopped ...

// Stop the client
asyncClient.stop();
```

### CallbackHandler Example
```java
class MyCallbackHandler extends NMClientAsync.AbstractCallbackHandler {
  @Override
  public void onContainerStarted(ContainerId containerId, Map<String, ByteBuffer> allServiceResponse) {
    // Post-process after the container is started, process the response
  }

  @Override
  public void onContainerResourceIncreased(ContainerId containerId, Resource resource) {
    // Post-process after the container resource is increased
  }

  @Override
  public void onContainerStatusReceived(ContainerId containerId, ContainerStatus containerStatus) {
    // Make use of the status of the container
  }

  @Override
  public void onContainerStopped(ContainerId containerId) {
    // Post-process after the container is stopped
  }

  @Override
  public void onStartContainerError(ContainerId containerId, Throwable t) {
    // Handle the raised exception
  }

  @Override
  public void onGetContainerStatusError(ContainerId containerId, Throwable t) {
    // Handle the raised exception
  }

  @Override
  public void onStopContainerError(ContainerId containerId, Throwable t) {
    // Handle the raised exception
  }
}
```
```

--------------------------------

### void initialize

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/azure/NativeAzureFileSystem.html

Initializes the FileSystem with a given URI and configuration.

```APIDOC
## void initialize(URI uri, Configuration conf)

### Description
Initialize a FileSystem with the provided URI and Configuration.

### Method
`initialize`

### Parameters
#### Path Parameters
- **uri** (URI) - The URI of the FileSystem.
- **conf** (Configuration) - The configuration object for the FileSystem.
```

--------------------------------

### FieldSelectionReducer setup Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/fieldsel/FieldSelectionReducer.html

The setup method is called once at the start of the task. It overrides the setup method from the Reducer class.

```java
public void setup(org.apache.hadoop.mapreduce.Reducer.Context context)
           throws IOException,
                  InterruptedException
```

--------------------------------

### Get Service Start Time

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/service/Service.html

Retrieves the time when the service was started. Returns zero if the service has not yet been started.

```Java
long getStartTime()

```

--------------------------------

### Get Node ID of Started Container

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Gets the NodeId of the node on which a container is running.

```APIDOC
## Get Node ID of Started Container

### Description
Get the NodeId of the node on which container is running.

### Method
N/A (This is a class/method documentation, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### setup Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJobBase.html

Static method to set up the job configuration. It takes a Configuration object as input.

```java
public static
void setup(Configuration
job)
```

--------------------------------

### setupJob

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/BindingPathOutputCommitter.html

Setup the job output during initialization.

```APIDOC
## setupJob(JobContext jobContext)

### Description
For the framework to setup the job output during initialization. This is called from the application master process for the entire job. This will be called multiple times, once per job attempt.

### Parameters
* `jobContext` (JobContext) - Required - Context of the job whose output is being written.

### Throws
* `IOException` - if temporary output could not be created
```

--------------------------------

### Get Successfully Started Containers

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/api/protocolrecords/StartContainersResponse.html

Retrieves the list of ContainerIds for containers that were successfully started. This is useful for confirming which containers are operational after a start request.

```java
@InterfaceAudience.Public
@InterfaceStability.Stable
public abstract
List<ContainerId>
getSuccessfullyStartedContainers()
```

--------------------------------

### Launch Example Service (CLI)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html

Command to launch a pre-built example service on YARN.

```bash
yarn app -launch <service-name> <example-name>

```

--------------------------------

### ValueAggregatorReducer Setup Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorReducer.html

Called once at the start of the task. Overrides the setup method from the Reducer class.

```java
public
void
setup(org.apache.hadoop.mapreduce.Reducer.Context
context)
           throws IOException,
                  InterruptedException
```

--------------------------------

### init Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/applications/distributedshell/Client.html

Parses command line options for client initialization.

```java
public
boolean
init(String[]
args)
             throws org.apache.commons.cli.ParseException
```

--------------------------------

### Get Start Time

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/JobStatus.html

Retrieves the start time of the job as a long timestamp. This indicates when the job execution began.

```Java
public long getStartTime()
```

--------------------------------

### FileSystem.initialize

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/conf/class-use/Configuration.html

Initialize a FileSystem.

```APIDOC
## FileSystem.initialize

### Description
Initialize a FileSystem.

### Method
void

### Endpoint
initialize(URI name, Configuration conf)

### Parameters
#### Path Parameters
- **name** (URI) - Required - The name of the FileSystem.
- **conf** (Configuration) - Required - The configuration object.

### Response
- No return value.
```

--------------------------------

### Get Job Status (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html

Example of an HTTP GET request to retrieve job status in JSON format.

```http
GET http://proxy-http-address:port/proxy/application_1326232085508_0004/ws/v1/mapreduce/jobs/job_1326232085508_4_4
```

```http
HTTP/1.1 200 OK
Content-Type: application/json
Server: Jetty(6.1.26)
Content-Length: 720
```

```json
{
   "job" : {
      "runningReduceAttempts" : 1,
      "reduceProgress" : 100,
      "failedReduceAttempts" : 0,
      "newMapAttempts" : 0,
      "mapsRunning" : 0,
      "state" : "RUNNING",
      "successfulReduceAttempts" : 0,
      "reducesRunning" : 1,
      "acls" : [
         {
            "value" : " ",
            "name" : "mapreduce.job.acl-modify-job"
         },
         {
            "value" : " ",
            "name" : "mapreduce.job.acl-view-job"
         }
      ],
      "reducesPending" : 0,
      "user" : "user1",
      "reducesTotal" : 1,
      "mapsCompleted" : 1,
      "startTime" : 1326238769379,
      "id" : "job_1326232085508_4_4",
      "successfulMapAttempts" : 1,
      "runningMapAttempts" : 0,
      "newReduceAttempts" : 0,
      "name" : "Sleep job",
      "mapsPending" : 0,
      "elapsedTime" : 59437,
      "reducesCompleted" : 0,
      "mapProgress" : 100,
      "diagnostics" : "",
      "failedMapAttempts" : 0,
      "killedReduceAttempts" : 0,
      "mapsTotal" : 1,
      "uberized" : false,
      "killedMapAttempts" : 0,
      "finishTime" : 0
   }
}
```

--------------------------------

### setupProgress

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/JobStatus.html

Returns the percentage of progress in setup.

```APIDOC
## setupProgress

### Description
Returns the percentage of progress in setup.

### Method
GET

### Endpoint
N/A (Method call)

### Returns
- **Percentage of progress in setup** (float) - The percentage of progress in setup.
```

--------------------------------

### Get Cluster Nodes (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Example HTTP GET request to retrieve cluster node information with an explicit 'Accept: application/xml' header to get the response in XML format.

```http
GET http://rm-http-address:port/ws/v1/cluster/nodes
Accept: application/xml
```

```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<nodes>
  <node>
    <rack>/default-rack</rack>
    <state>RUNNING</state>
    <id>host1.domain.com:54158</id>
    <nodeHostName>host1.domain.com</nodeHostName>
    <nodeHTTPAddress>host1.domain.com:8042</nodeHTTPAddress>
    <lastHealthUpdate>1476995346399</lastHealthUpdate>
    <version>3.0.0-SNAPSHOT</version>
    <healthReport></healthReport>
    <numContainers>0</numContainers>
    <usedMemoryMB>0</usedMemoryMB>
    <availMemoryMB>8192</availMemoryMB>
    <usedVirtualCores>0</usedVirtualCores>
    <availableVirtualCores>8</availableVirtualCores>
    <resourceUtilization>
        <nodePhysicalMemoryMB>1027</nodePhysicalMemoryMB>
        <nodeVirtualMemoryMB>1027</nodeVirtualMemoryMB>
        <nodeCPUUsage>0.006664445623755455</nodeCPUUsage>
        <aggregatedContainersPhysicalMemoryMB>0</aggregatedContainersPhysicalMemoryMB>
        <aggregatedContainersVirtualMemoryMB>0</aggregatedContainersVirtualMemoryMB>
        <containersCPUUsage>0.0</containersCPUUsage>
    </resourceUtilization>
  </node>
  <node>
    <rack>/default-rack</rack>
    <state>RUNNING</state>
    <id>host2.domain.com:54158</id>
    <nodeHostName>host2.domain.com</nodeHostName>
    <nodeHTTPAddress>host2.domain.com:8042</nodeHTTPAddress>
    <lastHealthUpdate>1476995346399</lastHealthUpdate>
    <version>3.0.0-SNAPSHOT</version>
    <healthReport></healthReport>
    <numContainers>0</numContainers>
    <usedMemoryMB>0</usedMemoryMB>
    <availMemoryMB>8192</availMemoryMB>
    <usedVirtualCores>0</usedVirtualCores>
    <availableVirtualCores>8</availableVirtualCores>
    <resourceUtilization>
        <nodePhysicalMemoryMB>1027</nodePhysicalMemoryMB>
        <nodeVirtualMemoryMB>1027</nodeVirtualMemoryMB>
        <nodeCPUUsage>0.006664445623755455</nodeCPUUsage>
        <aggregatedContainersPhysicalMemoryMB>0</aggregatedContainersPhysicalMemoryMB>
        <aggregatedContainersVirtualMemoryMB>0</aggregatedContainersVirtualMemoryMB>
        <containersCPUUsage>0.0</containersCPUUsage>
    </resourceUtilization>
  </node>
</nodes>
```

--------------------------------

### setupTask

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/FileOutputCommitter.html

Sets up output for the task.

```APIDOC
## setupTask(TaskAttemptContext context)

### Description
Sets up output for the task.

### Method
`public void setupTask(TaskAttemptContext context)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **context** (TaskAttemptContext) - Context of the task for which output is being set up.

### Throws
* `IOException`
```

--------------------------------

### long getStartTime()

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/Job.html

Gets the start time of the job.

```APIDOC
## getStartTime()

### Description
Get start time of the job.

### Returns
* **long** - the start time of the job.
```

--------------------------------

### Get Start Time of Allocation

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/api/records/ResourceAllocationRequest.html

Retrieves the start time for which the resource is allocated. This method is abstract and must be implemented by concrete subclasses.

```java
@InterfaceAudience.Public
@InterfaceStability.Unstable
public abstract long getStartTime()
```

--------------------------------

### initialize

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/azure/NativeAzureFileSystem.html

Initializes the FileSystem instance with a URI and configuration before it's ready for use.

```APIDOC
## initialize(URI uri, Configuration conf)

### Description
Initialize a FileSystem. Called after the new FileSystem instance is constructed, and before it is ready for use. FileSystem implementations overriding this method MUST forward it to their superclass, though the order in which it is done, and whether to alter the configuration before the invocation are options of the subclass.

### Method
`public void initialize(URI uri, Configuration conf)`

### Parameters
* **uri** (URI) - a URI whose authority section names the host, port, etc. for this FileSystem
* **conf** (Configuration) - the configuration

### Throws
* `IOException` - on any failure to initialize this instance.
* `IllegalArgumentException` - if the URI is considered invalid.
```

--------------------------------

### Get Application State (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Example of an HTTP GET request to retrieve the state of a specific application. The response is in XML format.

```HTTP
GET http://rm-http-address:port/ws/v1/cluster/apps/application_1399397633663_0003/state

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 99
Server: Jetty(6.1.26)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
  <state>ACCEPTED</state>
</appstate>
```

--------------------------------

### LocalFileSystem.initialize

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/conf/class-use/Configuration.html

Initializes the LocalFileSystem.

```APIDOC
## LocalFileSystem.initialize

### Description
Initializes the LocalFileSystem.

### Method
void

### Endpoint
initialize(URI name, Configuration conf)

### Parameters
#### Path Parameters
- **name** (URI) - Required - The name of the FileSystem.
- **conf** (Configuration) - Required - The configuration object.

### Response
- No return value.
```

--------------------------------

### Get Application State (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Example of an HTTP GET request to retrieve the state of a specific application. The response is in JSON format.

```HTTP
GET http://rm-http-address:port/ws/v1/cluster/apps/application_1399397633663_0003/state

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)

{
  "state":"ACCEPTED"
}
```

--------------------------------

### initialize

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/LocalFileSystem.html

Initializes the LocalFileSystem instance with a URI and configuration.

```APIDOC
## initialize(URI name, Configuration conf)

### Description
Called after a new FileSystem instance is constructed. Initializes the LocalFileSystem instance with a URI and configuration.

### Method
public void

### Parameters
* **name** (URI) - a uri whose authority section names the host, port, etc. for this FileSystem
* **conf** (Configuration) - the configuration

### Throws
* **IOException** - on any failure to initialize this instance.
```

--------------------------------

### setupTask

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.html

Performs setup for a task. No task setup is required by this committer.

```APIDOC
## setupTask

### Description
No task setup required.

### Parameters
#### Path Parameters
- **context** (TaskAttemptContext) - Required - Context of the task whose output is being written.

### Throws
- IOException
```

--------------------------------

### NMClientAsync Start Container

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/client/api/async/NMClientAsync.html

Asynchronously starts a container with the specified launch context.

```java
public abstract	void	startContainerAsync(Container	container,
                                         ContainerLaunchContext	containerLaunchContext)
```

--------------------------------

### FileOutputCommitter.setupJob

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/class-use/JobContext.html

Sets up the job, creating the temporary output directory.

```APIDOC
## FileOutputCommitter.setupJob

### Description
Create the temporary directory that is the root of all of the task work directories.

### Method
void

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### setup

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Performs setup operations. This method is available in ValueAggregatorJobBase (static), ValueAggregatorMapper, ValueAggregatorReducer, ChainMapper, ChainReducer, FieldSelectionMapper, FieldSelectionReducer, RegexMapper, Mapper, and Reducer classes.

```APIDOC
## setup(Configuration)

### Description
Performs setup operations.

### Method
Static method

### Endpoint
Not applicable (Java method)

### Parameters
- **Configuration** (Configuration) - Description not specified
```

```APIDOC
## setup(Mapper<K1, V1, Text, Text>.Context)

### Description
Performs setup operations for ValueAggregatorMapper.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Mapper<K1, V1, Text, Text>.Context** (Mapper<K1, V1, Text, Text>.Context) - Description not specified
```

```APIDOC
## setup(Reducer<Text, Text, Text, Text>.Context)

### Description
Performs setup operations for ValueAggregatorReducer.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Reducer<Text, Text, Text, Text>.Context** (Reducer<Text, Text, Text, Text>.Context) - Description not specified
```

```APIDOC
## setup(Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context)

### Description
Performs setup operations for ChainMapper.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context** (Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) - Description not specified
```

```APIDOC
## setup(Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context)

### Description
Performs setup operations for ChainReducer.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context** (Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) - Description not specified
```

```APIDOC
## setup(Mapper<K, Text, Text, LongWritable>.Context)

### Description
Performs setup operations for RegexMapper.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Mapper<K, Text, Text, LongWritable>.Context** (Mapper<K, Text, Text, LongWritable>.Context) - Description not specified
```

```APIDOC
## setup(Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context)

### Description
Called once at the beginning of the task for Mapper.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context** (Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) - Description not specified
```

```APIDOC
## setup(Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context)

### Description
Called once at the start of the task for Reducer.

### Method
Not specified (likely Java method call)

### Endpoint
Not applicable (Java method)

### Parameters
- **Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context** (Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) - Description not specified
```

--------------------------------

### newInstance(URI uri, Configuration conf, String user)

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html

Returns the FileSystem for a given URI, configuration, and user. This method internally calls `newInstance(URI, Configuration)` and is used to obtain a FileSystem instance associated with a specific user.

```APIDOC
## newInstance(URI uri, Configuration conf, String user)

### Description
Returns the FileSystem for this URI's scheme and authority and the given user. Internally invokes `newInstance(URI, Configuration)`.

### Method
static FileSystem

### Parameters
* **uri** (URI) - The URI of the filesystem.
* **conf** (Configuration) - The configuration to use.
* **user** (String) - The user to perform the get as.

### Returns
* filesystem instance.

### Throws
* `IOException` - if the FileSystem cannot be instantiated.
* `InterruptedException` - If the `UGI.doAs()` call was somehow interrupted.
```

--------------------------------

### Get Task Attempt Details (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html

Example HTTP GET request to retrieve details of a specific task attempt in JSON format.

```http
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
```

```http
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
```

```json
{
   "taskAttempt" : {
      "assignedContainerId" : "container_1326381300833_0002_01_000002",
      "progress" : 100,
      "elapsedTime" : 2638,
      "state" : "SUCCEEDED",
      "diagnostics" : "",
      "rack" : "/98.139.92.0",
      "nodeHttpAddress" : "host.domain.com:8042",
      "startTime" : 1326381450680,
      "id" : "attempt_1326381300833_2_2_m_0_0",
      "type" : "MAP",
      "finishTime" : 1326381453318
   }
}
```

--------------------------------

### TEMPLATE File Configuration Example

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/yarn-service/Configurations.html

Example of configuring a file using the TEMPLATE type, where content is directly provided.

```json
{
    "type": "TEMPLATE"
    "dest_file": "/etc/conf/hello"
    "properties" : {
         "content" : "Hello world"
    }
}

```

--------------------------------

### Get History Server Info (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html

Example HTTP GET request to retrieve general history server information in JSON format.

```http
GET http://history-server-http-address:port/ws/v1/history/info
```

```http
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
```

```json
{
   "historyInfo" : {
      "startedOn":1353512830963,
      "hadoopVersionBuiltOn" : "Wed Jan 11 21:18:36 UTC 2012",
      "hadoopBuildVersion" : "0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7",
      "hadoopVersion" : "0.23.1-SNAPSHOT"
   }
}
```

--------------------------------

### Get Job Status (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html

Example of an HTTP GET request with an Accept header for XML to retrieve job status in XML format.

```http
GET http://proxy-http-address:port/proxy/application_1326232085508_0004/ws/v1/mapreduce/jobs/job_1326232085508_4_4
Accept: application/xml
```

```http
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 1201
Server: Jetty(6.1.26)
```

```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<job>
  <startTime>1326238769379</startTime>
  <finishTime>0</finishTime>
  <elapsedTime>59474</elapsedTime>
  <id>job_1326232085508_4_4</id>
  <name>Sleep job</name>
  <user>user1</user>
  <state>RUNNING</state>
  <mapsTotal>1</mapsTotal>
  <mapsCompleted>1</mapsCompleted>
  <reducesTotal>1</reducesTotal>
  <reducesCompleted>0</reducesCompleted>
  <mapProgress>100.0</mapProgress>
  <reduceProgress>100.0</reduceProgress>
  <mapsPending>0</mapsPending>
  <mapsRunning>0</mapsRunning>
  <reducesPending>0</reducesPending>
  <reducesRunning>1</reducesRunning>
  <uberized>false</uberized>
  <diagnostics/>
  <newReduceAttempts>0</newReduceAttempts>
  <runningReduceAttempts>1</runningReduceAttempts>
  <failedReduceAttempts>0</failedReduceAttempts>
  <killedReduceAttempts>0</killedReduceAttempts>
  <successfulReduceAttempts>0</successfulReduceAttempts>
  <newMapAttempts>0</newMapAttempts>
  <runningMapAttempts>0</runningMapAttempts>
  <failedMapAttempts>0</failedMapAttempts>
  <killedMapAttempts>0</killedMapAttempts>
  <successfulMapAttempts>1</successfulMapAttempts>
  <acls>
    <name>mapreduce.job.acl-modify-job</name>
    <value> </value>
  </acls>
  <acls>
    <name>mapreduce.job.acl-view-job</name>    <value> </value>
  </acls>
</job>
```

--------------------------------

### ViewFsOverloadScheme Mount Link Configuration Example

Source: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html

Example configuration for ViewFsOverloadScheme showing how to map a logical mount link path to a target URI.

```xml
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://MyCluster1</value>
</property>

<property>
  <name>fs.viewfs.mounttable.MyCluster1./user</name>
  <value>hdfs://MyCluster2/user</value>
  <!-- mount table name : MyCluster1
       mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user
       mount link path: /user
       mount link uri: hdfs://MyCluster1/user
       mount target uri for /user: hdfs://MyCluster2/user -->
</property>
```

--------------------------------

### Partitioned Output Example

Source: https://hadoop.apache.org/docs/stable/hadoop-streaming/HadoopStreaming.html

Shows how map outputs are partitioned into reducers based on the first two key fields.

```text
11.11.4.1
-----------
11.12.1.2
11.12.1.1
-----------
11.14.2.3
11.14.2.2
```

--------------------------------

### Get Cluster Nodes (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Example HTTP GET request to retrieve cluster node information in JSON format. This is the default response format.

```http
GET http://rm-http-address:port/ws/v1/cluster/nodes
```

```json
{
  "nodes":
  {
    "node":
    [
      {
        "rack":"\/default-rack",
        "state":"RUNNING",
        "id":"host.domain.com:54158",
        "nodeHostName":"host.domain.com",
        "nodeHTTPAddress":"host.domain.com:8042",
        "lastHealthUpdate": 1476995346399,
        "version": "3.0.0-alpha2-SNAPSHOT",
        "healthReport":"",
        "numContainers":0,
        "usedMemoryMB":0,
        "availMemoryMB":8192,
        "usedVirtualCores":0,
        "availableVirtualCores":8,
        "resourceUtilization":
        {
          "nodePhysicalMemoryMB":1027,
          "nodeVirtualMemoryMB":1027,
          "nodeCPUUsage":0.016661113128066063,
          "aggregatedContainersPhysicalMemoryMB":0,
          "aggregatedContainersVirtualMemoryMB":0,
          "containersCPUUsage":0
        }
      },
      {
        "rack":"\/default-rack",
        "state":"RUNNING",
        "id":"host.domain.com:54158",
        "nodeHostName":"host.domain.com",
        "nodeHTTPAddress":"host.domain.com:8042",
        "lastHealthUpdate":1476995346399,
        "version":"3.0.0-alpha2-SNAPSHOT",
        "healthReport":"",
        "numContainers":0,
        "usedMemoryMB":0,
        "availMemoryMB":8192,
        "usedVirtualCores":0,
        "availableVirtualCores":8,
        "resourceUtilization":
        {
          "nodePhysicalMemoryMB":1027,
          "nodeVirtualMemoryMB":1027,
          "nodeCPUUsage":0.016661113128066063,
          "aggregatedContainersPhysicalMemoryMB":0,
          "aggregatedContainersVirtualMemoryMB":0,
          "containersCPUUsage":0
        }
      }
    ]
  }
}
```

--------------------------------

### Get Task Attempt State (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html

Example of an HTTP GET request to retrieve the state of a specific task attempt, with the response in XML format.

```http
GET http://proxy-http-address:port/proxy/application_1429692837321_0001/ws/v1/mapreduce/jobs/job_1429692837321_0001/tasks/task_1429692837321_0001_m_000000/attempts/attempt_1429692837321_0001_m_000000_0/state

HTTP/1.1 200 OK
Content-Type: application/xml
Server: Jetty(6.1.26)
Content-Length: 121

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobTaskAttemptState>
  <state>STARTING</state>
</jobTaskAttemptState>
```

--------------------------------

### Create and Check File Existence

Source: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/outputstream.html

Demonstrates the process of creating a file and verifying its existence and initial state.

```pseudocode
exists(FS, path)
(Stream', FS') = create(FS, path)
exists(FS', path)
getFileStatus(FS', path).getLen() = 0
```

--------------------------------

### Get Task Attempt State (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html

Example of an HTTP GET request to retrieve the state of a specific task attempt, with the response in JSON format.

```http
GET http://proxy-http-address:port/proxy/application_1429692837321_0001/ws/v1/mapreduce/jobs/job_1429692837321_0001/tasks/task_1429692837321_0001_m_000000/attempts/attempt_1429692837321_0001_m_000000_0/state

HTTP/1.1 200 OK
Content-Type: application/json
Server: Jetty(6.1.26)
Content-Length: 20

{
  "state":"STARTING"
}
```

--------------------------------

### main

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.html

Entry point for the ApplicationMaster. Parses command line arguments and starts the application.

```APIDOC
## main(String[] args)

### Description
Entry point for the ApplicationMaster. Parses command line arguments and starts the application.

### Method
public static void main(String[] args)

### Parameters
#### Path Parameters
- **args** (String[]) - Command line args
```

--------------------------------

### Get Attempts to Start Skipping

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/SkipBadRecords.html

Retrieves the number of task attempts after which skip mode is activated. This determines when the system starts reporting potentially bad records.

```java
public static
int getAttemptsToStartSkipping(Configuration conf)
```

--------------------------------

### Setup Java Environment

Source: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/UnixShellAPI.html

Configures or verifies the JAVA_HOME environment variable.

```shell
hadoop_java_setup
```

--------------------------------

### Job Setup Directories

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/manifest_committer_protocol.html

Creates the necessary directories for job operations.

```shell
mkdir(jobAttemptDirectory)
mkdir(manifestDirectory)
mkdir(taskAttemptDirectory)

```

--------------------------------

### Get Job Attempts (JSON)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html

Example of an HTTP GET request to retrieve job attempts in JSON format. The response includes details about each job attempt.

```http
GET http://proxy-http-address:port/proxy/application_1326232085508_0004/ws/v1/mapreduce/jobs/job_1326232085508_4_4/jobattempts

```

```http
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)

```

```json
{
   "jobAttempts" : {
      "jobAttempt" : [
         {
            "nodeId" : "host.domain.com:8041",
            "nodeHttpAddress" : "host.domain.com:8042",
            "startTime" : 1326238773493,
            "id" : 1,
            "logsLink" : "http://host.domain.com:8042/node/containerlogs/container_1326232085508_0004_01_000001",
            "containerId" : "container_1326232085508_0004_01_000001"
         }
      ]
   }
}

```

--------------------------------

### Initialize FileSystem

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html

Initializes a FileSystem.

```APIDOC
## initialize

### Description
Initialize a FileSystem.

### Method
void

### Endpoint
N/A (Instance method)

### Parameters
- name (URI): The URI of the FileSystem.
- conf (Configuration): The Hadoop configuration.

### Response
None
```

--------------------------------

### Get Offset Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/BlockLocation.html

Retrieves the starting offset of the block within the file.

```java
long getOffset()

```

--------------------------------

### setStartTime

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Sets the start time for various Hadoop entities like jobs, tasks, and reservations.

```APIDOC
## setStartTime

### Description
Sets the start time for various entities including jobs, tasks, and reservations.

### Method
Not specified (likely a Java method call).

### Endpoint
Not applicable (Java method).

### Parameters
* **long** or **Long** - Description of the start time value.
```

--------------------------------

### setupJob Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/FileOutputCommitter.html

Sets up the job output directory during initialization. This method is called by the framework for the entire job and can be invoked multiple times per job attempt.

```java
public void setupJob(JobContext context)
              throws IOException
```

--------------------------------

### Get Application Information

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html

This example demonstrates how to retrieve information about a specific application using an HTTP GET request. It shows a successful JSON response with application details.

```APIDOC
## GET /ws/v1/cluster/apps/application

### Description
Retrieves information about a specific application.

### Method
GET

### Endpoint
/ws/v1/cluster/apps/application

### Parameters
#### Query Parameters
- **applicationId** (string) - Required - The ID of the application to retrieve.

### Response
#### Success Response (200)
- **app** (object) - Contains details about the application.
  - **id** (string) - The unique identifier of the application.
  - **user** (string) - The user who submitted the application.
  - **name** (string) - The name of the application.
  - **queue** (string) - The queue the application belongs to.
  - **state** (string) - The current state of the application.
  - **finalStatus** (string) - The final status of the application.
  - **progress** (number) - The progress of the application.
  - **trackingUI** (string) - The tracking UI for the application.
  - **diagnostics** (string) - Diagnostic information about the application.
  - **clusterId** (number) - The ID of the cluster.
  - **startedTime** (number) - The time the application started.
  - **finishedTime** (number) - The time the application finished.
  - **elapsedTime** (number) - The elapsed time of the application.
  - **amContainerLogs** (string) - URL for the Application Master container logs.
  - **amHostHttpAddress** (string) - The HTTP address of the Application Master host.

#### Response Example
```json
{
  "app": {
    "id": "application_1324057493980_0001",
    "user": "user1",
    "name": "",
    "queue": "default",
    "state": "ACCEPTED",
    "finalStatus": "UNDEFINED",
    "progress": 0,
    "trackingUI": "UNASSIGNED",
    "diagnostics": "",
    "clusterId": 1324057493980,
    "startedTime": 1324057495921,
    "finishedTime": 0,
    "elapsedTime": 2063,
    "amContainerLogs": "http://amNM:2/node/containerlogs/container_1324057493980_0001_01_000001",
    "amHostHttpAddress": "amNM:2"
  }
}
```
```

--------------------------------

### startContainerAsync

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Starts an allocated container asynchronously. This method is part of the NMClientAsync class for managing NodeManager clients.

```APIDOC
## startContainerAsync

### Description
Starts an allocated container asynchronously.

### Method
Not specified (asynchronous method)

### Endpoint
Not applicable (SDK method)

### Parameters
- **Container** (Container) - Description not specified
- **ContainerLaunchContext** (ContainerLaunchContext) - Description not specified
```

--------------------------------

### Get History Server Info (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html

Example HTTP GET request to retrieve general history server information in XML format by setting the Accept header.

```http
GET http://history-server-http-address:port/ws/v1/history/info
Accept: application/xml
```

```http
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 330
Server: Jetty(6.1.26)
```

```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<historyInfo>
  <startedOn>1353512830963</startedOn>
  <hadoopVersion>0.23.1-SNAPSHOT</hadoopVersion>
  <hadoopBuildVersion>0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7</hadoopBuildVersion>
  <hadoopVersionBuiltOn>Wed Jan 11 21:18:36 UTC 2012</hadoopVersionBuiltOn>
</historyInfo>
```

--------------------------------

### setup Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorMapper.html

Called once at the beginning of the task. This method is an override of the setup method in the parent Mapper class.

```APIDOC
## setup(Mapper.Context context)

### Description
Called once at the beginning of the task.

### Method
`void setup(org.apache.hadoop.mapreduce.Mapper.Context context)`

### Overrides
`setup` in class `Mapper<K1 extends WritableComparable<?>,V1 extends Writable,Text,Text>`

### Throws
`IOException`
`InterruptedException`
```

--------------------------------

### Get Task Attempt Details (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html

Example HTTP GET request to retrieve details of a specific task attempt, requesting XML format via the Accept header.

```http
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
Accept: application/xml
```

```http
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 691
Server: Jetty(6.1.26)
```

```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<taskAttempt>
  <startTime>1326381450680</startTime>
  <finishTime>1326381453318</finishTime>
  <elapsedTime>2638</elapsedTime>
  <progress>100.0</progress>
  <id>attempt_1326381300833_2_2_m_0_0</id>
  <rack>/98.139.92.0</rack>
  <state>SUCCEEDED</state>
  <nodeHttpAddress>host.domain.com:8042</nodeHttpAddress>
  <diagnostics/>
  <type>MAP</type>
  <assignedContainerId>container_1326381300833_0002_01_000002</assignedContainerId>
</taskAttempt>
```

--------------------------------

### newInstance(URI uri, Configuration config)

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html

Creates and returns a new FileSystem object for the specified URI and configuration. The entire URI is used to initialize the FileSystem instance.

```APIDOC
## newInstance(URI uri, Configuration config)

### Description
Returns the FileSystem for this URI's scheme and authority. The entire URI is passed to the FileSystem instance's initialize method. This always returns a new FileSystem object.

### Method
static FileSystem

### Parameters
* **uri** (URI) - FS URI.
* **config** (Configuration) - Configuration to use.

### Returns
* the new FS instance.

### Throws
* `IOException` - FS creation or initialization failure.
```

--------------------------------

### setStartContainerRequests

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Sets a list of requests to start containers in YARN.

```APIDOC
## setStartContainerRequests

### Description
Sets a list of `StartContainerRequest` objects to initiate container startup.

### Method
Not specified (likely a Java method call).

### Endpoint
Not applicable (Java method).

### Parameters
* **List<StartContainerRequest>** - Description of the list of requests.
```

--------------------------------

### newInstance (URI, Configuration, User)

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html

Returns the FileSystem for the given URI's scheme and authority, for a specific user.

```APIDOC
## newInstance (URI, Configuration, User)

### Description
Returns the FileSystem for this URI's scheme and authority and the given user.

### Method
`newInstance`

### Parameters
#### URI Parameter
- **uri** (URI) - The URI of the filesystem.
#### Configuration Parameter
- **config** (Configuration) - The Hadoop configuration.
#### String Parameter
- **user** (String) - The user to impersonate.

### Return Value
`FileSystem` - A FileSystem instance.
```

--------------------------------

### Get All Applications (XML)

Source: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

This snippet shows an example HTTP GET request to retrieve all applications from the ResourceManager in XML format. It includes the request URL and expected response headers.

```http
GET http://rm-http-address:port/ws/v1/cluster/apps
Accept: application/xml
```

```http
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2459
Server: Jetty(6.1.26)
```

--------------------------------

### setupTask Method

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/OutputCommitter.html

Sets up output for an individual task. Called from each task's process, potentially multiple times for different task attempts.

```Java
public abstract	void	setupTask(TaskAttemptContext	taskContext)
                        throws IOException

```

--------------------------------

### getUnmanagedAM

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html

Gets if the RM should manage the execution of the AM. If true, then the RM will not allocate a container for the AM and start it.

```APIDOC
## getUnmanagedAM

### Description
Gets if the RM should manage the execution of the AM. If true, then the RM will not allocate a container for the AM and start it. It will expect the AM to be launched and connect to the RM within the AM liveliness period and fail the app otherwise. The client should launch the AM only after the RM has ACCEPTED the application and changed the `YarnApplicationState`. Such apps will not be retried by the RM on app attempt failure. The default value is false.

### Returns
true if the AM is not managed by the RM
```

--------------------------------

### BindingPathOutputCommitter.setupJob

Source: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/class-use/JobContext.html

Sets up the job for this committer.

```APIDOC
## BindingPathOutputCommitter.setupJob

### Description
Sets up the job for this committer.

### Method
void

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### setResetTimeOnStart

Source: https://hadoop.apache.org/docs/stable/api/index-all.html

Configures whether to reset the timer upon starting.

```APIDOC
## setResetTimeOnStart boolean

### Method
Method in class org.apache.hadoop.yarn.util.AbstractLivelinessMonitor
```
Tokens:
Raw