2 distance between the original and reconstructed images forces the encoder to learn a compressed representation of the input. The training set was collected by running a random policy on the simplified task described in the previous section. Since they are not relevant for our task, we filtered out the plane and gripper fingers from the images. Figure~\ref{fig:encoder_results} shows two examples of original, filtered, reconstructed and error images using an encoder trained on a  dataset of 50000 images, using 120 epochs of the Adam optimizer~\cite{kingma2014adam}, with a learning rate of $2\cdot 10^{-4}$, and batch size of $128$. The same encoder weights were used throughout all experiments in this work.

\begin{figure}
  \centering
  \includegraphics[width=0.7\columnwidth]{figures/encoder_results}
  \caption{Two samples of images processed by our perception pipeline. From left to right, we show simulated RGB and filtered depth images, reconstructions produced by the autoencoder and the difference between originals and reconstructions.}
  \label{fig:encoder_results}
\end{figure}

\subsubsection*{Control Module} 

We use a small network that is trained separately from the perception network to map encoded observations to optimal actions.
Policies are modeled as multivariate Gaussian distributions.
A feed-forward neural network with two hidden layers and ReLU activations maps observations to the means of the distribution while the log-standard deviations are parameterized by a global, trainable vector.
Actions are normalized to the range of $[-1, 1]$ using a $\tanh$ output non-linearity.
Policy weights are optimized using \gls{trpo}~\cite{schulman2015trust}, a policy gradient method that performs stable updates by enforcing a constraint on the maximum change in policy distributions between two updates.

\subsection{Simulation}

Collecting data using a dynamic simulation and synthetic depth images instead of a real system has several advantages: it is faster, scales better, has lower cost, there is no need for supervision, automatic reset of experiments is easy to implement, and full state information is available.
For this reason, we focused on performing all training in simulation.
We constructed a virtual world using the Bullet physics engine~\cite{coumans2018} and added a disentangled robot hand whose position is controlled via a force constraint, avoiding the computation of inverse kinematics.
A virtual camera rendering $64 \times 64$ images was placed to match the viewpoint of the real setup. Depth images were generated using a software-renderer bundled with the physics engine and filtering was performed using masks provided by the engine.

\subsection{Transfer to the Real Platform}

We explore transferring policies trained in simulation to the real world without any fine-tuning of the network weights.
Ideally, images captured from the real camera would only need to be resized and cropped to match the dimensions of the simulated camera and then be passed into the encoder.
However, due to imperfect data and high noise levels, especially at the operating boundaries of the real sensor, some additional filtering was required.
In particular, we noticed increasing noise and some curvature towards image boundaries, as well as high noise around the gripper's fingers.
For this reason, we applied an additional elliptic mask to filter out the borders and dilated masks of the gripper's fingers.
The surface was detected and filtered using a \gls{ransac}~\cite{fischler1981random} based approach.


\section{Evaluation}\label{sec:evaluation}

The goal of our experiments is to evaluate and compare training times and final performance of the proposed models, as well as assess their capability to react to dynamic changes and transfer to the real world.

\subsection{Experimental Setup}

The platform used for evaluation consists of a position controlled 7-\gls{dof} arm of an ABB Yumi with a maximal payload of \SI{250}{g}.
The fingers of a stock gripper with opening width of \SI{5}{cm} were rubber-covered for better grip and reducing reflection.
A CamBoard pico flexx time of flight camera was attached to the wrist of the robot at a tilt angle of \SI{15}{\degree} as seen in the top right image of Figure~\ref{fig:teaser_image}.
In simulation, we used a model that matches the real robot and step the dynamics simulation with a size of \SI{5}{ms} which provided plausible physical behavior.
Training was performed on a set of procedurally generated random objects with diverse shapes\footnote{\url{https://sites.google.com/site/brainrobotdata/home/models}}.
Following \cite{quillen2018deep}, we split the dataset into 900 train and 100 test models and the objects were scaled to fit into the smaller gripper used in this work.
The grasping task was implemented on top of the OpenAI gym interface~\cite{brockman2016openai} and we based our implementation of \gls{trpo} on Rllab~\cite{duan2016benchmarking}.
Policy iterations were performed using a step size of $10^{-2}$ and a batch size of $10^4$ and $2 \cdot 10^4$ for the simplified/full task description respectively.
A curriculum of eight sets of workspace parameters was used with values linearly increasing in the ranges reported in Table~\ref{table:curriculum_parameters}.
\revisedtext{The curriculum step $\lambda$} is increased once a threshold success rate $\epsilon{=}0.7$ averaged over the last 1000 episodes was reached during training.

\begin{table}
  \caption{Parameter ranges for the curriculum sequences.}
  \centering
  \begin{tabular}{l|rr}
    \toprule
    {\bfseries Parameter} & {\bfseries Min. value} & {\bfseries Max. value} \\
    \hline
    $l$ & \SI{2}{cm} & \SI{20}{cm} \\
    $h_\text{robot}$ & \SI{4}{cm} & \SI{16}{cm} \\
    $h_\text{lift}$ & \SI{1}{cm} & \SI{10}{cm} \\
    $n_\text{max}$ & 3 & 5 \\
    \bottomrule
  \end{tabular}
  \label{table:curriculum_parameters}
\end{table}

\subsection{Simulated Experiments}

\subsubsection{Model comparison}

\begin{figure}
  \centering
  \includegraphics[width=\columnwidth]{figures/model_comparison.png}
  \caption{Learning curves for the different models analyzed in this work. Using a curriculum significantly speeds up training and leads to high final success rates that are comparable to the performance of an agent which was trained with a warm start provided by a grasping heuristic with fewer \gls{dof}.}
  \label{fig:model_comparison}
\end{figure}

\revisedtext{We analyze learning curves and the final performance of models trained on the full problem with only shaped rewards (\textit{shaped}), and using the proposed curriculum with both shaped and sparse reward formulations (\textit{shaped/sparse, curriculum}).
We also include agents trained on the simplified task (\textit{sparse, simplified}), the \gls{bc} (\textit{sparse, \gls{bc}}) and warm-started policies (\textit{sparse, warm-start}) described in Section~\ref{sec:transfer_learning}.}
Figure~\ref{fig:model_comparison} shows success rates of training iterations against \revisedtext{the number of environment interactions}.
For each model, we performed experiments with five different seeds and report the \revisedtext{median}, as well as the worst and best run, depicted as solid lines and shaded areas respectively.
Surprisingly, even when training on the full workspace from the start, the algorithm manages to reinforce the occasional intermediate reward provided when agents interact with objects.
However, results strongly varied over the seeds, with three out of the five runs failing altogether.
Using a curriculum significantly speeds up learning as well as the final performance of the agents.
We observe that the difference in the learning curves using the shaped and sparse rewards are quite small.
This confirms that providing easily reachable goal states at early stages of training acts as a mean of guiding the agent and speeding up the training process without artificially shaping the reward function.
Note that both of these models seem to stagnate temporally around \revisedtext{steps 1 to $3 \cdot 10^6$}.
This is due to the agents repeatedly reaching a success rate of $0.7$, triggering an increase in difficulty of the task until the curriculum parameters are set to their maximum values.
\revisedtext{This behavior is depicted in more details in Figure~\ref{fig:curriculum_study}(c), plotting the history of the curriculum step $\lambda$ along the history of success rates.}

\revisedtext{We can also observe that replacing two \gls{dof} with a heuristic (\textit{sparse, simplified}) results in a task that is considerably easier and faster to learn.
However, policies seem to converge to a lower success rate.
We found that a large fraction of these failure cases are due to a collision check, implemented to avoid the agent wasting time in case the robot stalled