Examples of using L1General

These demos have been set-up to show how L1General can be used to optimize a variety of smooth objective functions subject to L1-regularization.

The code used to run these demos can be downloaded here (which includes a copy of the 2010 version of L1General). Note that several of the examples require that Graphviz is installed for plotting graphs.

To run all of the demos, type (in Matlab):

>> cd L1GeneralExamples      % Change to the extracted directory
>> addpath(genpath(pwd))     % Add all sub-directories to the Matlab path
>> mexAll                    % Compile mex files (not necessary on all systems)
>> L1General_examples        % Run the examples

Contents

LASSO

% Generate some data
nInstances = 250;
nVars = 50;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);

% Least Squares Solution
wLS = X\y;

% Ridge Regression
lambda = 100*ones(nVars,1); % Penalize each element by the same amount
R = chol(X'*X + diag(lambda));
wRR = R\(R'\(X'*y));

% LASSO
lambda = 100*ones(nVars,1); % Penalize the absolute value of each element by the same amount
funObj = @(w)SquaredError(w,X,y); % Loss function that L1 regularization is applied to
w_init = wRR; % Initial value for iterative optimizer
fprintf('\nComputing LASSO Coefficients...\n');
wLASSO = L1General2_PSSgb(funObj,w_init,lambda);

fprintf('Number of non-zero variables in Least Squares solution: %d\n',nnz(wLS));
fprintf('Number of non-zero variables in Ridge Regression solution: %d\n',nnz(wRR));
fprintf('Number of non-zero variables in LASSO solution: %d\n',nnz(wLASSO));

figure;
clf;hold on;
subplot(2,2,1);
stem(wLS,'r');
xlim([1 nVars]);
yl = ylim;
title('Least Squares');
subplot(2,2,2);
stem(wRR,'b');
xlim([1 nVars]);
ylim(yl);
title('Ridge Regression');
subplot(2,2,3);
stem(wLASSO,'g');
xlim([1 nVars]);
title('LASSO');
ylim(yl);
pause;
Computing LASSO Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 2.49592e-04 1.99301e+03 3.16398e+02     43
     2      3 1.00000e+00 1.76918e+03 8.91349e+01     19
     3      4 1.00000e+00 1.75367e+03 3.12464e+01     21
     4      5 1.00000e+00 1.75160e+03 4.64930e+00     21
     5      6 1.00000e+00 1.75145e+03 8.46870e-01     20
     6      7 1.00000e+00 1.75145e+03 2.53280e-01     20
     7      8 1.00000e+00 1.75145e+03 9.39831e-02     20
     8      9 1.00000e+00 1.75145e+03 1.55213e-02     20
     9     10 1.00000e+00 1.75145e+03 5.85301e-03     20
    10     11 1.00000e+00 1.75145e+03 8.86230e-04     20
    11     12 1.00000e+00 1.75145e+03 9.06708e-04     20
Progress in parameters or objective below progTol
Number of non-zero variables in Least Squares solution: 50
Number of non-zero variables in Ridge Regression solution: 50
Number of non-zero variables in LASSO solution: 20

Elastic Net

lambdaL2 = 100*ones(nVars,1);
lambdaL1 = 100*ones(nVars,1);
penalizedFunObj = @(w)penalizedL2(w,funObj,lambdaL2);
fprintf('\nComputing Elastic Net Coefficients...\n');
wElastic = L1General2_PSSgb(penalizedFunObj,w_init,lambdaL1);

fprintf('Number of non-zero variables in Elastic Net solution: %d\n',nnz(wElastic));

subplot(2,2,4);
stem(wElastic,'c');
xlim([1 nVars]);
ylim(yl);
title('Elastic Net');
pause;
Computing Elastic Net Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 2.00000e-04 3.27151e+03 9.28964e+01     43
     2      3 1.00000e+00 3.12917e+03 2.52775e+01     22
     3      4 1.00000e+00 3.12721e+03 1.26982e+01     22
     4      5 1.00000e+00 3.12675e+03 1.25912e+00     21
     5      6 1.00000e+00 3.12674e+03 2.57555e-01     21
     6      7 1.00000e+00 3.12674e+03 1.07619e-01     21
     7      8 1.00000e+00 3.12674e+03 1.64976e-02     21
     8      9 1.00000e+00 3.12674e+03 1.06693e-03     21
     9     10 1.00000e+00 3.12674e+03 4.13000e-04     21
    10     11 1.00000e+00 3.12674e+03 7.03746e-05     21
Progress in parameters or objective below progTol
Number of non-zero variables in Elastic Net solution: 21

Logistic Regression

X = [ones(nInstances,1) X]; % Add Bias element to features
y = sign(y); % Convert y to binary {-1,1} representation

funObj = @(w)LogisticLoss(w,X,y);
w_init = zeros(nVars+1,1);

% Maximum Likelihood
fprintf('\nComputing Maximum Likelihood Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wLogML = minFunc(funObj,w_init,mfOptions);

% L2-Regularized Logistic Regression
fprintf('\nComputing L2-Regularized Logistic Regression Coefficients...\n');
lambda = 15*ones(nVars+1,1);
lambda(1) = 0; % Do not penalize bias variable
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wLogL2 = minFunc(funObjL2,w_init,mfOptions);

% L1-Regularized Logistic Regression
fprintf('\nComputing L1-Regularized Logistic Regression Coefficients...\n');
wLogL1 = L1General2_PSSgb(funObj,w_init,lambda);

% Elastic Net Logistic Regression
fprintf('\nComputing Elastic-Net Logistic Regression Coefficients...\n');
wLogL1L2 = L1General2_PSSgb(funObjL2,w_init,lambda);

figure;
clf;hold on;
subplot(2,2,1);
stem(wLogML,'r');
xlim([1 nVars+1]);
title('Maximum Likelihood Logistic Regression');
subplot(2,2,2);
stem(wLogL2,'b');
xlim([1 nVars+1]);
title('L2-Regularized Logistic Regression');
subplot(2,2,3);
stem(wLogL1,'g');
xlim([1 nVars+1]);
title('L1-Regularized Logistic Regression');
subplot(2,2,4);
stem(wLogL1L2,'c');
xlim([1 nVars+1]);
title('Elastic-Net Logistic Regression');

fprintf('Number of Features Selected by Maximum Likelihood Logistic Regression classifier: %d (out of %d)\n',nnz(wLogML(2:end)),nVars);
fprintf('Number of Features Selected by L2-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL2(2:end)),nVars);
fprintf('Number of Features Selected by L1-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1(2:end)),nVars);
fprintf('Number of Features Selected by Elastic-Net Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1L2(2:end)),nVars);
fprintf('Classification error rate on training data for L1-regularied Logistic Regression: %.2f\n',sum(y ~= sign(X*wLogL1))/length(y));
pause;
Computing Maximum Likelihood Logistic Regression Coefficients
 Iteration   FunEvals     Step Length    Function Val        Opt Cond
         1          2     1.00000e+00     7.07045e+01     1.97507e+01
         2          3     1.00000e+00     4.23053e+01     8.07759e+00
         3          4     1.00000e+00     2.66959e+01     3.49396e+00
         4          5     1.00000e+00     1.62143e+01     1.55590e+00
         5          6     1.00000e+00     8.10661e+00     6.90238e-01
         6          7     1.00000e+00     3.38291e+00     2.82238e-01
         7          8     1.00000e+00     1.25949e+00     1.07228e-01
         8          9     1.00000e+00     4.61854e-01     3.98042e-02
         9         10     1.00000e+00     1.69851e-01     1.47850e-02
        10         11     1.00000e+00     6.26252e-02     5.48087e-03
        11         12     1.00000e+00     2.31072e-02     2.02741e-03
        12         13     1.00000e+00     8.52514e-03     7.48738e-04
        13         14     1.00000e+00     3.14421e-03     2.76176e-04
        14         15     1.00000e+00     1.15922e-03     1.01770e-04
        15         16     1.00000e+00     4.27240e-04     3.74723e-05
        16         17     1.00000e+00     1.57415e-04     1.37887e-05
        17         18     1.00000e+00     5.79822e-05     5.07135e-06
Optimality Condition below optTol

Computing L2-Regularized Logistic Regression Coefficients...
 Iteration   FunEvals     Step Length    Function Val        Opt Cond
         1          2     1.00000e+00     1.05577e+02     8.83408e+00
         2          3     1.00000e+00     1.03449e+02     5.21616e-01
         3          4     1.00000e+00     1.03438e+02     3.09434e-03
         4          5     1.00000e+00     1.03438e+02     1.26932e-07
Optimality Condition below optTol

Computing L1-Regularized Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 6.00892e-03 1.50194e+02 3.29826e+01     11
     2      3 1.00000e+00 1.35832e+02 8.96380e+00      7
     3      4 1.00000e+00 1.34305e+02 2.60035e+00      7
     4      5 1.00000e+00 1.34139e+02 3.92810e-01      7
     5      6 1.00000e+00 1.34136e+02 1.24296e-01      7
     6      7 1.00000e+00 1.34135e+02 6.21160e-02      7
     7      8 1.00000e+00 1.34135e+02 1.12322e-03      7
     8      9 1.00000e+00 1.34135e+02 1.24925e-04      7
     9     10 1.00000e+00 1.34135e+02 8.38783e-06      7
First-order optimality below optTol

Computing Elastic-Net Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 6.00892e-03 1.52889e+02 2.33081e+01     11
     2      3 1.00000e+00 1.48589e+02 2.77799e+00      8
     3      4 1.00000e+00 1.48514e+02 2.03804e-01      8
     4      5 1.00000e+00 1.48514e+02 1.74764e-02      8
     5      6 1.00000e+00 1.48514e+02 3.36673e-03      8
     6      7 1.00000e+00 1.48514e+02 1.50247e-04      8
Directional derivative below progTol
Number of Features Selected by Maximum Likelihood Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L2-regualrized Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L1-regualrized Logistic Regression classifier: 6 (out of 50)
Number of Features Selected by Elastic-Net Logistic Regression classifier: 7 (out of 50)
Classification error rate on training data for L1-regularied Logistic Regression: 0.10

L-BFGS Variant

nVars = 5000;

X = [ones(nInstances,1) randn(nInstances,nVars-1)];
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1));

lambda = 10*ones(nVars,1);
lambda(1) = 0;
funObj = @(w)LogisticLoss(w,X,y);
fprintf('\nComputing Logistic Regression Coefficients for model with %d variables with L-BFGS\n',nVars);
wLogistic = L1General2_PSSgb(funObj,zeros(nVars,1),lambda);

figure;
clf;
stem(wLogistic,'g');
title(sprintf('L1-Regularized Logistic Regression (%d vars)',nVars));

fprintf('Number of Features Selected by Logistic Regression classifier: %d (out of %d)\n',nnz(wLogistic(2:end)),nVars);
fprintf('Classification error rate on training data: %.2f\n',sum(y ~= sign(X*wLogistic))/length(y));
pause;
Computing Logistic Regression Coefficients for model with 5000 variables with L-BFGS
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 2.24113e-04 1.67068e+02 1.96701e+01   1147
     2      3 1.00000e+00 1.48643e+02 8.84668e+00    768
     3      4 1.00000e+00 1.45242e+02 6.51621e+00    583
     4      5 1.00000e+00 1.42914e+02 4.75094e+00    386
     5      6 1.00000e+00 1.39937e+02 3.44001e+00    312
     6      7 1.00000e+00 1.36860e+02 2.99006e+00    191
     7      8 1.00000e+00 1.36225e+02 4.07078e+00    122
Backtracking...
Backtracking...
Backtracking...
Backtracking...
     8     13 6.59677e-04 1.36204e+02 3.92474e+00    125
     9     14 1.00000e+00 1.35018e+02 2.49947e+00    128
    10     15 1.00000e+00 1.34552e+02 1.19953e+00    120
    11     16 1.00000e+00 1.34424e+02 1.19847e+00    110
    12     17 1.00000e+00 1.34295e+02 6.10298e-01    106
Backtracking...
    13     19 8.03893e-03 1.34287e+02 6.31270e-01    109
    14     20 1.00000e+00 1.34267e+02 1.10060e+00    106
    15     21 1.00000e+00 1.34223e+02 3.80897e-01    101
    16     22 1.00000e+00 1.34209e+02 1.68550e-01    102
    17     23 1.00000e+00 1.34203e+02 1.52216e-01    101
    18     24 1.00000e+00 1.34201e+02 1.75575e-01    100
    19     25 1.00000e+00 1.34199e+02 1.29820e-01     99
    20     26 1.00000e+00 1.34199e+02 6.71145e-02     99
Backtracking...
    21     28 3.08890e-01 1.34198e+02 3.54430e-02     99
    22     29 1.00000e+00 1.34198e+02 2.49175e-02     99
    23     30 1.00000e+00 1.34198e+02 1.44747e-02     99
    24     31 1.00000e+00 1.34198e+02 2.46634e-02     99
    25     32 1.00000e+00 1.34198e+02 3.25376e-02     98
    26     33 1.00000e+00 1.34198e+02 4.11185e-03     99
    27     34 1.00000e+00 1.34198e+02 2.29290e-03     99
    28     35 1.00000e+00 1.34198e+02 2.41097e-03     99
    29     36 1.00000e+00 1.34198e+02 1.80338e-03     99
    30     37 1.00000e+00 1.34198e+02 4.79784e-04     99
    31     38 1.00000e+00 1.34198e+02 2.82431e-04     99
    32     39 1.00000e+00 1.34198e+02 1.58254e-04     99
    33     40 1.00000e+00 1.34198e+02 1.12707e-04     99
    34     41 1.00000e+00 1.34198e+02 1.27808e-04     99
    35     42 1.00000e+00 1.34198e+02 3.38671e-05     99
Directional derivative below progTol
Number of Features Selected by Logistic Regression classifier: 98 (out of 5000)
Classification error rate on training data: 0.00

Lasso Regularization Path

% Generate some data
nInstances = 100;
nVars = 10;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);

% Find Maximum value for regularization parameter and pick step increment
[f,g] = SquaredError(zeros(nVars,1),X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Least Squares L1-Regularization path\n');
funObj = @(w)SquaredError(w,X,y);
w = zeros(nVars,1);
options = struct('verbose',0);
for mult = 1-lambdaInc:-lambdaInc:0
    lambda = mult*lambdaMax*ones(nVars,1);
    w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;xData = 1:-lambdaInc:0;
yData = w;
plot(xData,yData);
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Least Squares)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
Computing Least Squares L1-Regularization path

Huber Robust Regression Regularization Path

changePoint = 1;

% Find Maximum value for regularization parameter and pick step increment
[f,g] = HuberLoss(zeros(nVars,1),X,y,changePoint);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Huber Robust Regression L1-Regularization path\n');
funObj = @(w)HuberLoss(w,X,y,changePoint);
w = zeros(nVars,1);
options = struct('verbose',0);
for mult = 1-lambdaInc:-lambdaInc:0
    lambda = mult*lambdaMax*ones(nVars,1);
    w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;xData = 1:-lambdaInc:0;
yData = w;
plot(xData,yData);
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Huber Robust Regression)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
Computing Huber Robust Regression L1-Regularization path

Logistic Regression Regularization Path

X = [ones(nInstances,1) X]; % Add Bias element to features
y = sign(y); % Convert y to binary {-1,1} representation

% Solve for Maximum Likelihood bias element
funObjBias = @(w)LogisticLoss(w,ones(nInstances,1),y);
bias = minFunc(funObjBias,0,struct('Display','none'));

% Find Maximum value for regularization parameter and pick step increment
w = [bias;zeros(nVars,1)];
[f,g] = LogisticLoss(w,X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Logistic Regression L1-Regularization path\n');
funObj = @(w)LogisticLoss(w,X,y);
for mult = 1-lambdaInc:-lambdaInc:lambdaInc
    lambda = [0;mult*lambdaMax*ones(nVars,1)];
    w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;
plot(1:-lambdaInc:lambdaInc,w');
legend({'Bias'});
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Logistic Regression)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
Computing Logistic Regression L1-Regularization path

Probit Regression, Smooth SVM, Huberized SVM

% Generate some data
nInstances = 500;
nVars = 100;
X = randn(nInstances,nVars);
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1)/5);

% Add Unpenalized Bias
X = [ones(nInstances,1) X];
lambda = [0;10*ones(nVars,1)];

fprintf('\nComputing Logistic Regression Coefficients...\n');
funObj = @(w)LogisticLoss(w,X,y);
wLogit = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Probit Regression Coefficients...\n');
funObj = @(w)ProbitLoss(w,X,y);
wProbit = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Smooth Support Vector Machine Coefficients...\n');
funObj = @(w)SSVMLoss(w,X,y);
wSSVM = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Huberized Support Vector Machine Coefficients...\n');
t = .5;
funObj = @(w)HuberSVMLoss(w,X,y,t);
wHSVM = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Extreme Value Regression Coefficients...\n');
funObj = @(w)ExtremeLoss(w,X,y);
wExtreme = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

figure;
clf;hold on;
subplot(3,2,1);
stem(wLogit,'r');
xlim([1 nVars]);
title('L1-Regularized Logistic Regression');
subplot(3,2,2);
stem(wProbit,'b');
xlim([1 nVars]);
title('L1-Regularized Probit Regression');
subplot(3,2,3);
stem(wSSVM,'g');
xlim([1 nVars]);
title('L1-Regularized Smooth SVM');
% subplot(3,2,4);
% stem(max(0,1-y.*(X*wSSVM)),'c');
% xlim([1 nInstances]);
% title('Hinge Loss for L1-Regularized SSVM');
subplot(3,2,4);
stem(wSSVM,'k');
xlim([1 nVars]);
title('L1-Regularized Huberized SVM');
subplot(3,2,5);
stem(max(0,1-y.*(X*wHSVM)),'m');
xlim([1 nInstances]);
title('Hinge Loss for L1-Regularized HSVM');
subplot(3,2,6)
stem(wExtreme,'c');
xlim([1 nVars]);
title('L1-Regularized Extreme-Value Regression');

fprintf('Number of non-zero variables for Logistic Regression: %d out of %d\n',nnz(wLogit(2:end)),nVars);
fprintf('Number of non-zero variables for Probit Regression:  %d out of %d\n',nnz(wProbit(2:end)),nVars);
fprintf('Number of non-zero variables for Smooth Support Vector Machine: %d out of %d',nnz(wSSVM(2:end)),nVars);
fprintf(' (%d support vectors)\n',sum(1-y.*(X*wSSVM)>=0));
fprintf('Number of non-zero variables for Huberized Support Vector Machine: %d out of %d',nnz(wHSVM(2:end)),nVars);
fprintf(' (%d support vectors)\n',sum(1-y.*(X*wHSVM)>=0));
fprintf('Number of non-zero variables for Extreme Value Regression: %d out of %d\n',nnz(wExtreme(2:end)),nVars);
pause;
Computing Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 1.11219e-03 3.16670e+02 5.24024e+01     57
     2      3 1.00000e+00 2.35672e+02 1.43651e+01     41
     3      4 1.00000e+00 2.25972e+02 5.97437e+00     39
     4      5 1.00000e+00 2.23176e+02 2.36338e+00     39
     5      6 1.00000e+00 2.23072e+02 2.01425e+00     36
     6      7 1.00000e+00 2.22938e+02 3.12517e-01     38
     7      8 1.00000e+00 2.22937e+02 8.83875e-02     38
     8      9 1.00000e+00 2.22937e+02 1.42327e-02     38
     9     10 1.00000e+00 2.22937e+02 4.08680e-03     38
    10     11 1.00000e+00 2.22937e+02 2.92593e-03     38
    11     12 1.00000e+00 2.22937e+02 3.79409e-04     38
    12     13 1.00000e+00 2.22937e+02 1.28940e-04     38
    13     14 1.00000e+00 2.22937e+02 2.42499e-05     38
Progress in parameters or objective below progTol

Computing Probit Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 5.54304e-04 3.00238e+02 8.39334e+01     70
     2      3 1.00000e+00 2.02146e+02 2.79057e+01     55
     3      4 1.00000e+00 1.84718e+02 1.19909e+01     54
     4      5 1.00000e+00 1.78376e+02 7.11523e+00     52
     5      6 1.00000e+00 1.77394e+02 5.00142e+00     47
     6      7 1.00000e+00 1.77010e+02 1.53044e+00     48
     7      8 1.00000e+00 1.76994e+02 4.22121e-01     49
     8      9 1.00000e+00 1.76991e+02 1.09574e-01     49
     9     10 1.00000e+00 1.76990e+02 4.26942e-02     49
    10     11 1.00000e+00 1.76990e+02 2.76763e-02     49
    11     12 1.00000e+00 1.76990e+02 7.00188e-03     49
    12     13 1.00000e+00 1.76990e+02 2.48059e-03     49
    13     14 1.00000e+00 1.76990e+02 1.13110e-03     49
    14     15 1.00000e+00 1.76990e+02 3.87316e-04     49
Backtracking...
    15     17 4.11044e-01 1.76990e+02 9.68925e-05     49
Directional derivative below progTol

Computing Smooth Support Vector Machine Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 1.75512e-04 3.84887e+02 2.17640e+02     89
     2      3 1.00000e+00 1.85515e+02 5.45835e+01     79
     3      4 1.00000e+00 1.63225e+02 2.68508e+01     84
     4      5 1.00000e+00 1.47825e+02 1.83658e+01     72
     5      6 1.00000e+00 1.44710e+02 1.31008e+01     75
     6      7 1.00000e+00 1.43129e+02 4.64824e+00     76
     7      8 1.00000e+00 1.42845e+02 3.32625e+00     74
     8      9 1.00000e+00 1.42638e+02 2.47059e+00     73
     9     10 1.00000e+00 1.42542e+02 2.05454e+00     68
    10     11 1.00000e+00 1.42527e+02 1.42891e+00     68
    11     12 1.00000e+00 1.42512e+02 2.77524e-01     69
    12     13 1.00000e+00 1.42511e+02 1.86336e-01     69
    13     14 1.00000e+00 1.42510e+02 1.03574e-01     69
    14     15 1.00000e+00 1.42510e+02 8.98808e-02     69
    15     16 1.00000e+00 1.42510e+02 2.62123e-02     69
    16     17 1.00000e+00 1.42510e+02 1.28982e-02     69
    17     18 1.00000e+00 1.42510e+02 6.27322e-03     69
    18     19 1.00000e+00 1.42510e+02 7.70964e-03     69
    19     20 1.00000e+00 1.42510e+02 1.39856e-03     69
    20     21 1.00000e+00 1.42510e+02 1.04135e-03     69
    21     22 1.00000e+00 1.42510e+02 7.66865e-04     69
    22     23 1.00000e+00 1.42510e+02 6.20470e-04     69
    23     24 1.00000e+00 1.42510e+02 1.37625e-04     69
Directional derivative below progTol

Computing Huberized Support Vector Machine Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 4.09809e-04 3.09573e+02 1.37617e+02     76
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
     2      8 1.82115e-03 1.80320e+02 4.68808e+01     70
Backtracking...
     3     10 1.90768e-01 1.70621e+02 3.96830e+01     69
     4     11 1.00000e+00 1.44292e+02 1.89085e+01     65
     5     12 1.00000e+00 1.42175e+02 2.90943e+01     56
     6     13 1.00000e+00 1.36127e+02 9.61605e+00     58
     7     14 1.00000e+00 1.34907e+02 4.21485e+00     61
     8     15 1.00000e+00 1.34341e+02 3.61168e+00     59
     9     16 1.00000e+00 1.34096e+02 3.43694e+00     61
    10     17 1.00000e+00 1.33976e+02 2.31866e+00     61
    11     18 1.00000e+00 1.33938e+02 9.81897e-01     60
    12     19 1.00000e+00 1.33921e+02 5.69325e-01     61
    13     20 1.00000e+00 1.33913e+02 4.59229e-01     61
    14     21 1.00000e+00 1.33909e+02 3.31591e-01     61
    15     22 1.00000e+00 1.33907e+02 2.17188e-01     61
    16     23 1.00000e+00 1.33907e+02 1.01368e-01     60
    17     24 1.00000e+00 1.33907e+02 4.39434e-02     60
    18     25 1.00000e+00 1.33907e+02 3.06268e-02     60
    19     26 1.00000e+00 1.33907e+02 3.10286e-02     60
    20     27 1.00000e+00 1.33907e+02 1.04722e-02     60
    21     28 1.00000e+00 1.33907e+02 8.10877e-03     60
    22     29 1.00000e+00 1.33907e+02 3.55788e-03     60
    23     30 1.00000e+00 1.33907e+02 1.75012e-03     60
    24     31 1.00000e+00 1.33907e+02 1.35978e-03     60
Backtracking...
    25     33 4.67586e-01 1.33907e+02 9.97574e-04     60
    26     34 1.00000e+00 1.33907e+02 2.80376e-04     60
    27     35 1.00000e+00 1.33907e+02 1.71551e-04     60
    28     36 1.00000e+00 1.33907e+02 1.31008e-04     60
Progress in parameters or objective below progTol

Computing Extreme Value Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 5.40408e-04 3.13060e+02 7.98254e+01     66
     2      3 1.00000e+00 2.23061e+02 2.81775e+01     57
     3      4 1.00000e+00 1.99555e+02 1.32625e+01     52
     4      5 1.00000e+00 1.90739e+02 6.44873e+00     50
     5      6 1.00000e+00 1.89194e+02 4.24375e+00     47
     6      7 1.00000e+00 1.88798e+02 1.24892e+00     46
     7      8 1.00000e+00 1.88766e+02 4.46984e-01     47
     8      9 1.00000e+00 1.88762e+02 1.16921e-01     47
     9     10 1.00000e+00 1.88761e+02 6.50265e-02     47
    10     11 1.00000e+00 1.88761e+02 3.61242e-02     47
    11     12 1.00000e+00 1.88761e+02 4.43847e-03     47
    12     13 1.00000e+00 1.88761e+02 1.99794e-03     47
    13     14 1.00000e+00 1.88761e+02 8.04958e-04     47
    14     15 1.00000e+00 1.88761e+02 8.22021e-04     47
    15     16 1.00000e+00 1.88761e+02 1.02701e-04     47
Directional derivative below progTol
Number of non-zero variables for Logistic Regression: 37 out of 100
Number of non-zero variables for Probit Regression:  48 out of 100
Number of non-zero variables for Smooth Support Vector Machine: 68 out of 100 (246 support vectors)
Number of non-zero variables for Huberized Support Vector Machine: 59 out of 100 (256 support vectors)
Number of non-zero variables for Extreme Value Regression: 46 out of 100

Non-Parametric Logistic Regression with Sparse Prototypes

% Generate Non-Linear Data Set
nInstances = 100;
nVars = 2;
nExamplePoints = 4; % Set to 1 for linear classifier, higher for more non-linear
examplePoints = randn(nExamplePoints,nVars);
X = randn(nInstances,nVars);
y = zeros(nInstances,1);
for i = 1:nInstances
    dists = sum((repmat(X(i,:),nExamplePoints,1) - examplePoints).^2,2);
    [minVal minInd] = min(dists);
    y(i,1) = sign(mod(minInd,2)-.5);
end

% Make Gram Matrix
XX = kernelRBF(X,X,1);

fprintf('Computing Non-Parametric Logistic Regression Coefficients...\n');
funObj = @(u)LogisticLoss(u,XX,y);
lambda = .5*ones(nInstances,1);
u = L1General2_PSSgb(funObj,zeros(nInstances,1),lambda);

% Plot Data
fprintf('Generating Plot...\n');
increment = 100;
figure;
clf; hold on;
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
domainx = xlim;
domain1 = domainx(1):(domainx(2)-domainx(1))/increment:domainx(2);
domainy = ylim;
domain2 = domainy(1):(domainy(2)-domainy(1))/increment:domainy(2);
d1 = repmat(domain1',[1 length(domain1)]);
d2 = repmat(domain2,[length(domain2) 1]);
vals = sign(kernelRBF([d1(:) d2(:)],X,1)*u);
zData = reshape(vals,size(d1));
contourf(d1,d2,zData+rand(size(zData))/1000,[-1 0],'k');
colormap([0 0 .5;0 .5 0]);
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
xlim(domainx);
ylim(domainy);

% Circle Prototypes
prototypes = X(u~=0,:);
fprintf('(%d prototypes)\n',size(prototypes,1));
h=plot(prototypes(:,1),prototypes(:,2),'ro');
set(h,'MarkerSize',10);
legend(h,'Non-Zero Data Points (Prototypes)');
pause;
Computing Non-Parametric Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 5.20093e-03 6.67109e+01 3.98997e+00     95
     2      3 1.00000e+00 4.58075e+01 2.30255e+00     87
     3      4 1.00000e+00 3.94334e+01 1.18437e+00     95
     4      5 1.00000e+00 3.76799e+01 5.71931e-01     94
     5      6 1.00000e+00 3.69351e+01 4.62534e-01     81
     6      7 1.00000e+00 3.65222e+01 3.60834e-01     74
     7      8 1.00000e+00 3.60562e+01 2.90577e-01     60
     8      9 1.00000e+00 3.56919e+01 3.29955e-01     58
     9     10 1.00000e+00 3.51743e+01 5.80422e-01     39
Backtracking...
    10     12 4.94147e-01 3.50166e+01 3.76194e-01     57
    11     13 1.00000e+00 3.49203e+01 3.51194e-01     48
Backtracking...
    12     15 4.24323e-01 3.48336e+01 5.50364e-01     43
    13     16 1.00000e+00 3.47961e+01 3.82925e-01     38
Backtracking...
    14     18 1.53452e-01 3.47810e+01 3.44684e-01     43
    15     19 1.00000e+00 3.47560e+01 2.37006e-01     40
    16     20 1.00000e+00 3.47112e+01 1.84156e-01     39
Backtracking...
    17     22 3.77444e-01 3.46402e+01 2.80544e-01     36
Backtracking...
Backtracking...
    18     25 1.52675e-03 3.46398e+01 1.26413e-01     34
    19     26 1.00000e+00 3.46247e+01 1.34099e-01     31
    20     27 1.00000e+00 3.45933e+01 8.93053e-02     32
Backtracking...
    21     29 3.51940e-01 3.45605e+01 1.55374e-01     28
    22     30 1.00000e+00 3.45233e+01 1.27413e-01     30
    23     31 1.00000e+00 3.44337e+01 1.40339e-01     26
    24     32 1.00000e+00 3.43791e+01 6.37444e-02     26
    25     33 1.00000e+00 3.43549e+01 6.82287e-02     26
    26     34 1.00000e+00 3.43109e+01 7.09055e-02     24
Backtracking...
    27     36 4.64589e-01 3.42888e+01 7.98909e-02     24
    28     37 1.00000e+00 3.42771e+01 7.27030e-02     22
Backtracking...
    29     39 2.66997e-01 3.42700e+01 6.56971e-02     22
    30     40 1.00000e+00 3.42575e+01 3.44141e-02     20
    31     41 1.00000e+00 3.42456e+01 4.57312e-02     17
Backtracking...
    32     43 1.74912e-01 3.42423e+01 1.72429e-02     18
Backtracking...
    33     45 3.82482e-01 3.42399e+01 2.25954e-02     17
    34     46 1.00000e+00 3.42365e+01 3.08340e-02     18
Backtracking...
    35     48 1.54622e-01 3.42267e+01 4.95427e-02     17
Backtracking...
    36     50 4.83985e-01 3.42238e+01 1.37321e-02     16
    37     51 1.00000e+00 3.42193e+01 1.86491e-02     17
    38     52 1.00000e+00 3.42045e+01 3.49788e-02     16
    39     53 1.00000e+00 3.41973e+01 6.28513e-02     15
Backtracking...
    40     55 1.63954e-01 3.41943e+01 4.28451e-02     17
    41     56 1.00000e+00 3.41907e+01 2.04922e-02     17
    42     57 1.00000e+00 3.41870e+01 2.52734e-02     19
    43     58 1.00000e+00 3.41833e+01 2.02741e-02     15
Backtracking...
    44     60 4.34681e-01 3.41824e+01 1.19566e-02     14
    45     61 1.00000e+00 3.41817e+01 7.43078e-03     14
    46     62 1.00000e+00 3.41803e+01 4.93236e-03     13
Backtracking...
    47     64 5.58898e-01 3.41734e+01 9.05689e-03     12
Backtracking...
    48     66 3.42166e-01 3.41725e+01 2.68754e-02     12
    49     67 1.00000e+00 3.41677e+01 4.04584e-02     14
Backtracking...
    50     69 9.78283e-02 3.41658e+01 1.14057e-02     15
Backtracking...
    51     71 1.88171e-01 3.41655e+01 3.55383e-03     11
    52     72 1.00000e+00 3.41654e+01 1.08598e-02     11
    53     73 1.00000e+00 3.41653e+01 7.74173e-03     11
    54     74 1.00000e+00 3.41651e+01 3.52344e-03     11
    55     75 1.00000e+00 3.41645e+01 8.91291e-03     11
    56     76 1.00000e+00 3.41631e+01 2.06320e-02     12
Backtracking...
    57     78 2.31946e-01 3.41601e+01 1.94931e-02     12
Backtracking...
    58     80 6.76889e-03 3.41599e+01 1.95370e-02     12
Backtracking...
    59     82 4.16411e-03 3.41598e+01 1.95078e-02     12
Backtracking...
    60     84 2.71201e-03 3.41597e+01 1.94916e-02     12
Backtracking...
    61     86 1.99485e-03 3.41595e+01 1.94838e-02     12
Backtracking...
    62     88 1.78948e-03 3.41594e+01 1.94696e-02     12
Backtracking...
    63     90 1.47365e-03 3.41593e+01 1.94257e-02     12
Backtracking...
    64     92 1.50794e-03 3.41591e+01 1.93989e-02     12
Backtracking...
    65     94 1.37584e-03 3.41590e+01 1.93798e-02     12
Backtracking...
    66     96 1.31671e-03 3.41589e+01 1.93498e-02     12
Backtracking...
    67     98 1.24712e-03 3.41587e+01 1.93265e-02     12
Backtracking...
    68    100 1.20677e-03 3.41586e+01 1.92994e-02     12
Backtracking...
    69    102 1.15896e-03 3.41584e+01 1.92750e-02     12
Backtracking...
    70    104 1.12421e-03 3.41583e+01 1.92489e-02     12
Backtracking...
    71    106 1.08777e-03 3.41581e+01 1.92243e-02     12
Backtracking...
    72    108 1.06066e-03 3.41580e+01 1.91987e-02     12
Backtracking...
    73    110 1.02772e-03 3.41578e+01 1.91742e-02     12
Backtracking...
    74    112 1.00146e-03 3.41577e+01 1.91494e-02     12
Backtracking...
    75    114 1.00000e-03 3.41575e+01 1.91245e-02     12
Backtracking...
    76    116 1.00000e-03 3.41574e+01 1.90990e-02     12
Backtracking...
    77    118 1.00000e-03 3.41572e+01 1.90733e-02     12
Backtracking...
    78    120 1.00000e-03 3.41570e+01 1.90470e-02     12
Backtracking...
    79    122 1.00000e-03 3.41568e+01 1.90205e-02     12
Backtracking...
    80    124 1.00000e-03 3.41566e+01 1.89936e-02     12
Backtracking...
    81    126 1.00000e-03 3.41564e+01 1.89665e-02     12
Backtracking...
    82    128 1.00000e-03 3.41561e+01 1.29535e-02     11
    83    129 1.00000e+00 3.41538e+01 1.23784e-02     14
Backtracking...
Backtracking...
Backtracking...
    84    133 5.50929e-05 3.41538e+01 1.25504e-02     13
    85    134 1.00000e+00 3.41534e+01 2.51666e-02     10
Backtracking...
    86    136 1.37384e-03 3.41533e+01 2.47699e-02     15
Backtracking...
    87    138 3.51700e-03 3.41528e+01 1.76441e-02     13
Backtracking...
    88    140 1.89601e-01 3.41517e+01 7.73706e-03     12
Backtracking...
    89    142 3.37197e-03 3.41517e+01 9.62912e-03     12
Backtracking...
    90    144 2.97847e-01 3.41515e+01 1.64100e-02     11
    91    145 1.00000e+00 3.41511e+01 6.18655e-03     12
Backtracking...
Backtracking...
    92    148 1.93642e-02 3.41511e+01 5.83796e-03     10
Backtracking...
    93    150 8.28119e-02 3.41511e+01 7.42889e-03      9
    94    151 1.00000e+00 3.41509e+01 1.07153e-02      9
    95    152 1.00000e+00 3.41505e+01 7.41690e-03     11
Backtracking...
Backtracking...
    96    155 2.93548e-02 3.41505e+01 7.50382e-03     11
Backtracking...
Backtracking...
    97    158 8.68140e-03 3.41504e+01 9.12714e-03     10
Backtracking...
    98    160 1.40617e-01 3.41504e+01 7.55935e-03     11
    99    161 1.00000e+00 3.41499e+01 1.03130e-02     12
Backtracking...
Backtracking...
Backtracking...
   100    165 8.82333e-03 3.41499e+01 1.01387e-02     10
Backtracking...
   101    167 1.00000e-03 3.41489e+01 6.75874e-03     11
Backtracking...
Backtracking...
   102    170 2.65946e-04 3.41489e+01 7.65777e-03     10
   103    171 1.00000e+00 3.41486e+01 6.94239e-03     12
Backtracking...
   104    173 1.00000e-03 3.41484e+01 4.77170e-03     12
   105    174 1.00000e+00 3.41479e+01 4.86944e-03     12
   106    175 1.00000e+00 3.41476e+01 3.78948e-03     12
   107    176 1.00000e+00 3.41462e+01 6.57536e-03     12
Backtracking...
   108    178 5.87849e-01 3.41448e+01 1.01789e-02     11
Backtracking...
   109    180 2.22710e-02 3.41447e+01 1.02974e-02     11
Backtracking...
   110    182 1.75794e-02 3.41444e+01 1.50959e-02     10
   111    183 1.00000e+00 3.41438e+01 1.17165e-02     13
Backtracking...
Backtracking...
   112    186 1.00000e-06 3.41438e+01 9.29967e-03     13
Backtracking...
   113    188 1.00000e-03 3.41437e+01 8.84492e-03     13
Backtracking...
   114    190 1.89882e-01 3.41437e+01 1.61095e-02     12
Backtracking...
Backtracking...
   115    193 1.00000e-06 3.41434e+01 1.21053e-02     13
Backtracking...
Backtracking...
   116    196 1.92639e-04 3.41434e+01 7.50063e-03     11
Backtracking...
   117    198 4.55396e-02 3.41434e+01 5.90399e-03     12
Backtracking...
Backtracking...
   118    201 2.18368e-03 3.41434e+01 5.12003e-03     11
Backtracking...
   119    203 3.39055e-01 3.41433e+01 6.91099e-03     11
   120    204 1.00000e+00 3.41432e+01 5.68694e-03     11
   121    205 1.00000e+00 3.41429e+01 4.05702e-03     11
   122    206 1.00000e+00 3.41425e+01 2.49618e-03     10
Backtracking...
Backtracking...
Backtracking...
   123    210 7.54404e-06 3.41425e+01 3.39387e-03     11
Backtracking...
   124    212 8.71145e-02 3.41424e+01 3.24684e-03     10
Backtracking...
   125    214 4.12030e-03 3.41424e+01 3.01374e-03     10
   126    215 1.00000e+00 3.41424e+01 4.33100e-03     10
Backtracking...
   127    217 3.99388e-02 3.41424e+01 4.80645e-03     10
Backtracking...
Backtracking...
   128    220 1.37085e-03 3.41424e+01 4.14702e-03     10
Backtracking...
   129    222 7.43227e-02 3.41423e+01 2.64810e-03      9
Backtracking...
   130    224 3.96939e-01 3.41422e+01 1.36154e-03      9
   131    225 1.00000e+00 3.41422e+01 2.45462e-03     10
Backtracking...
   132    227 6.40876e-03 3.41422e+01 1.29458e-03     10
   133    228 1.00000e+00 3.41422e+01 6.06514e-04      9
   134    229 1.00000e+00 3.41421e+01 1.68009e-03     10
   135    230 1.00000e+00 3.41420e+01 2.64087e-03     10
Backtracking...
   136    232 4.93517e-01 3.41419e+01 8.76492e-03      9
Backtracking...
   137    234 2.96325e-01 3.41418e+01 4.52922e-03     12
Backtracking...
Backtracking...
   138    237 7.01401e-03 3.41418e+01 2.71449e-03     11
Backtracking...
Backtracking...
   139    240 1.76529e-04 3.41418e+01 3.18220e-03     10
   140    241 1.00000e+00 3.41417e+01 1.71625e-03     10
Backtracking...
   141    243 3.07791e-01 3.41417e+01 2.92124e-03      9
Backtracking...
   142    245 1.41743e-01 3.41416e+01 2.41884e-03      9
Backtracking...
   143    247 1.75011e-03 3.41416e+01 2.41427e-03      9
Backtracking...
   144    249 1.09712e-03 3.41416e+01 2.40656e-03      9
Backtracking...
   145    251 1.02424e-03 3.41416e+01 2.40346e-03      9
Backtracking...
   146    253 1.00000e-03 3.41416e+01 2.39969e-03      9
Backtracking...
   147    255 1.00000e-03 3.41416e+01 2.39668e-03      9
Backtracking...
   148    257 1.00000e-03 3.41416e+01 2.39354e-03      9
Backtracking...
   149    259 1.00000e-03 3.41416e+01 2.39055e-03      9
Backtracking...
   150    261 1.00000e-03 3.41416e+01 2.38754e-03      9
Backtracking...
   151    263 1.00000e-03 3.41416e+01 2.38455e-03      9
Backtracking...
   152    265 1.00000e-03 3.41416e+01 2.38156e-03      9
Backtracking...
   153    267 1.00000e-03 3.41416e+01 2.37858e-03      9
Backtracking...
   154    269 1.00000e-03 3.41416e+01 2.37559e-03      9
Backtracking...
   155    271 1.00000e-03 3.41416e+01 2.37260e-03      9
Backtracking...
   156    273 1.00000e-03 3.41416e+01 2.36961e-03      9
Backtracking...
   157    275 1.00000e-03 3.41416e+01 2.36663e-03      9
Backtracking...
   158    277 1.00000e-03 3.41416e+01 2.36364e-03      9
Backtracking...
   159    279 1.00000e-03 3.41416e+01 2.36065e-03      9
Backtracking...
   160    281 1.00000e-03 3.41416e+01 2.35765e-03      9
Backtracking...
   161    283 1.00000e-03 3.41416e+01 2.35466e-03      9
Backtracking...
   162    285 1.00000e-03 3.41416e+01 2.35167e-03      9
Backtracking...
   163    287 1.00000e-03 3.41416e+01 2.34867e-03      9
Backtracking...
   164    289 1.00000e-03 3.41416e+01 2.34568e-03      9
Backtracking...
   165    291 1.00000e-03 3.41416e+01 2.34268e-03      9
Backtracking...
   166    293 1.00000e-03 3.41416e+01 2.33969e-03      9
Backtracking...
   167    295 1.00000e-03 3.41416e+01 2.33669e-03      9
Backtracking...
   168    297 1.00000e-03 3.41416e+01 2.33369e-03      9
Backtracking...
   169    299 1.00000e-03 3.41416e+01 2.33069e-03      9
Backtracking...
Backtracking...
   170    302 6.00000e-04 3.41416e+01 2.32889e-03      9
Backtracking...
Backtracking...
   171    305 7.84761e-05 3.41416e+01 2.32866e-03      9
Backtracking...
Backtracking...
   172    308 6.65533e-05 3.41416e+01 2.32846e-03      9
Backtracking...
Backtracking...
Backtracking...
   173    312 8.75293e-06 3.41416e+01 2.32843e-03      9
Backtracking...
Backtracking...
Backtracking...
   174    316 5.64212e-06 3.41416e+01 2.32841e-03      9
Backtracking...
Backtracking...
Backtracking...
   175    320 4.52019e-06 3.41416e+01 2.32840e-03      9
Backtracking...
Backtracking...
Backtracking...
   176    324 3.87492e-06 3.41416e+01 2.32839e-03      9
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
   177    330 9.55608e-08 3.41416e+01 2.32839e-03      9
Progress in parameters or objective below progTol
Generating Plot...
(9 prototypes)

Multinomial Logistic Regression

% Generate Data
nInstances = 200;
nVars = 10;
nClasses = 5;
X = [ones(nInstances,1) randn(nInstances,nVars-1)];
w = randn(nVars,nClasses-1).*(rand(nVars,nClasses-1)>.5);
[junk y] = max(X*[w zeros(nVars,1)],[],2);

% Initialize Weights and Objective Function
w_init = zeros(nVars,nClasses-1);
w_init = w_init(:);
funObj = @(w)SoftmaxLoss2(w,X,y,nClasses);

% Set up regularizer
lambda = 1*ones(nVars,nClasses-1);
lambda(1,:) = 0; % Don't regularize bias elements
lambda = lambda(:);

% Maximum Likelihood
fprintf('\nComputing Maximum Likelihood Multinomial Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wMLR_ML = minFunc(funObj,w_init,mfOptions);
wMLR_ML = reshape(wMLR_ML,nVars,nClasses-1);

% L2-Regularized Logistic Regression
fprintf('\nComputing L2-Regularized Multinomial Logistic Regression Coefficients...\n');
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wMLR_L2 = minFunc(funObjL2,w_init,mfOptions);
wMLR_L2 = reshape(wMLR_L2,nVars,nClasses-1);

% L1-Regularized Logistic Regression
fprintf('\nComputing L1-Regularized Multinomial Logistic Regression Coefficients...\n');
wMLR_L1 = L1General2_PSSgb(funObj,w_init,lambda);
wMLR_L1 = reshape(wMLR_L1,nVars,nClasses-1);

% Elastic Net Logistic Regression
fprintf('\nComputing Elastic-Net Multinomial Logistic Regression Coefficients...\n');
wMLR_L1L2 = L1General2_PSSgb(funObjL2,w_init,lambda);
wMLR_L1L2 = reshape(wMLR_L1L2,nVars,nClasses-1);

% Report Number of non-zeros
fprintf('Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_ML(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L2-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L2(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L1-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by Elastic-Net Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1L2(2:end,:)),(nVars-1)*(nClasses-1));

% Show Stem Plots
figure;
clf;hold on;
subplot(2,2,1);
stem(wMLR_ML(:),'r');
title('Maximum Likelihood Multinomial Logistic');
subplot(2,2,2);
stem(wMLR_L2(:),'b');
title('L2-Regularized Multinomial Logistic');
subplot(2,2,3);
stem(wMLR_L1(:),'g');
title('L1-Regularized Multinomial Logistic');
subplot(2,2,4);
stem(wMLR_L1L2(:),'c');
title('Elastic-Net Multinomial Logistic');

% Compute training error
[junk yhat] = max(X*[wMLR_L1 zeros(nVars,1)],[],2);
fprintf('Classification error rate on training data for L1-regularied Multinomial Logistic: %.2f\n',sum(y ~= yhat)/length(y));
pause;
Computing Maximum Likelihood Multinomial Logistic Regression Coefficients
 Iteration   FunEvals     Step Length    Function Val        Opt Cond
         1          2     1.00000e+00     1.04179e+02     1.17219e+01
         2          3     1.00000e+00     6.22764e+01     5.13364e+00
         3          4     1.00000e+00     3.91936e+01     2.21929e+00
         4          5     1.00000e+00     2.42730e+01     1.13250e+00
         5          6     1.00000e+00     1.46139e+01     6.12513e-01
         6          7     1.00000e+00     7.94355e+00     2.90802e-01
         7          8     1.00000e+00     3.59489e+00     1.22797e-01
         8          9     1.00000e+00     1.38213e+00     4.79815e-02
         9         10     1.00000e+00     5.10257e-01     1.80193e-02
        10         11     1.00000e+00     1.88486e-01     6.82512e-03
        11         12     1.00000e+00     6.96973e-02     2.55603e-03
        12         13     1.00000e+00     2.57607e-02     9.51627e-04
        13         14     1.00000e+00     9.51241e-03     3.52916e-04
        14         15     1.00000e+00     3.50930e-03     1.30531e-04
        15         16     1.00000e+00     1.29366e-03     4.81926e-05
        16         17     1.00000e+00     4.76609e-04     1.77716e-05
        17         18     1.00000e+00     1.75517e-04     6.54829e-06
Optimality Condition below optTol

Computing L2-Regularized Multinomial Logistic Regression Coefficients...
 Iteration   FunEvals     Step Length    Function Val        Opt Cond
         1          2     1.00000e+00     1.20545e+02     1.02673e+01
         2          3     1.00000e+00     9.92603e+01     2.54050e+00
         3          4     1.00000e+00     9.63988e+01     4.25870e-01
         4          5     1.00000e+00     9.63028e+01     2.67172e-02
         5          6     1.00000e+00     9.63026e+01     1.06566e-04
         6          7     1.00000e+00     9.63026e+01     1.55572e-09
Optimality Condition below optTol

Computing L1-Regularized Multinomial Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 1.76103e-03 2.94845e+02 5.21182e+01     37
     2      3 1.00000e+00 1.35657e+02 1.26400e+01     35
     3      4 1.00000e+00 1.16746e+02 7.63398e+00     37
     4      5 1.00000e+00 9.17691e+01 6.58194e+00     35
     5      6 1.00000e+00 8.30630e+01 7.05694e+00     35
     6      7 1.00000e+00 7.90554e+01 3.16541e+00     36
     7      8 1.00000e+00 7.73113e+01 1.38246e+00     32
     8      9 1.00000e+00 7.60217e+01 1.01551e+00     32
     9     10 1.00000e+00 7.53268e+01 1.12175e+00     30
    10     11 1.00000e+00 7.49766e+01 1.13828e+00     30
    11     12 1.00000e+00 7.47120e+01 4.41812e-01     29
    12     13 1.00000e+00 7.45730e+01 4.17875e-01     29
    13     14 1.00000e+00 7.44940e+01 3.95614e-01     29
    14     15 1.00000e+00 7.44419e+01 3.31846e-01     29
    15     16 1.00000e+00 7.44120e+01 1.84727e-01     28
    16     17 1.00000e+00 7.44051e+01 8.31556e-02     29
    17     18 1.00000e+00 7.44018e+01 4.36642e-02     27
    18     19 1.00000e+00 7.43996e+01 3.15859e-02     27
    19     20 1.00000e+00 7.43993e+01 3.41207e-02     27
    20     21 1.00000e+00 7.43991e+01 1.57060e-02     27
    21     22 1.00000e+00 7.43990e+01 1.24550e-02     27
    22     23 1.00000e+00 7.43989e+01 8.23950e-03     27
    23     24 1.00000e+00 7.43989e+01 1.04371e-02     27
    24     25 1.00000e+00 7.43989e+01 2.68026e-03     27
    25     26 1.00000e+00 7.43989e+01 1.90409e-03     27
    26     27 1.00000e+00 7.43989e+01 1.70540e-03     27
    27     28 1.00000e+00 7.43988e+01 9.66254e-04     27
    28     29 1.00000e+00 7.43988e+01 1.11360e-03     27
    29     30 1.00000e+00 7.43988e+01 2.63725e-04     27
    30     31 1.00000e+00 7.43988e+01 1.19382e-04     27
    31     32 1.00000e+00 7.43988e+01 3.08955e-05     27
    32     33 1.00000e+00 7.43988e+01 2.96448e-05     27
Progress in parameters or objective below progTol

Computing Elastic-Net Multinomial Logistic Regression Coefficients...
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 1.76103e-03 2.94888e+02 5.19179e+01     37
     2      3 1.00000e+00 1.44710e+02 1.13796e+01     35
     3      4 1.00000e+00 1.31208e+02 6.10043e+00     37
     4      5 1.00000e+00 1.20146e+02 4.62499e+00     35
     5      6 1.00000e+00 1.18592e+02 4.70310e+00     38
     6      7 1.00000e+00 1.17639e+02 8.48525e-01     36
     7      8 1.00000e+00 1.17521e+02 6.80493e-01     37
     8      9 1.00000e+00 1.17378e+02 5.87050e-01     33
     9     10 1.00000e+00 1.17330e+02 1.68602e-01     31
    10     11 1.00000e+00 1.17324e+02 1.67430e-01     32
    11     12 1.00000e+00 1.17321e+02 7.02801e-02     32
    12     13 1.00000e+00 1.17320e+02 5.55746e-02     32
    13     14 1.00000e+00 1.17320e+02 3.14440e-02     32
    14     15 1.00000e+00 1.17320e+02 3.21882e-02     32
    15     16 1.00000e+00 1.17319e+02 1.95754e-02     32
Backtracking...
    16     18 3.59348e-01 1.17319e+02 1.42054e-02     32
    17     19 1.00000e+00 1.17319e+02 9.01286e-03     32
    18     20 1.00000e+00 1.17319e+02 4.78039e-03     32
    19     21 1.00000e+00 1.17319e+02 3.84352e-03     32
Backtracking...
    20     23 4.22797e-01 1.17319e+02 3.22387e-03     32
    21     24 1.00000e+00 1.17319e+02 7.25821e-04     32
    22     25 1.00000e+00 1.17319e+02 4.46354e-04     32
    23     26 1.00000e+00 1.17319e+02 2.49745e-04     32
Backtracking...
    24     28 4.83418e-01 1.17319e+02 1.18676e-04     32
    25     29 1.00000e+00 1.17319e+02 3.63166e-05     32
Directional derivative below progTol
Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L2-regualrized Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L1-regualrized Multinomial Logistic classifier: 23 (out of 36)
Number of Features Selected by Elastic-Net Multinomial Logistic classifier: 28 (out of 36)
Classification error rate on training data for L1-regularied Multinomial Logistic: 0.03

Compressed Sensing

figure;
nVars = 256;
nNonZero = 5;
x = zeros(nVars,1);
x(ceil(rand(nNonZero,1)*nVars)) = 1;

subplot(2,2,1);
imagesc(reshape(x,[16 16]));
colormap gray
title(sprintf('Original Signal (%d elements, %d nonZero)',nVars,nNonZero));

nMeasurements = 49;
phi = rand(nMeasurements,nVars);

subplot(2,2,2);
imagesc(phi);
colormap gray
title('Measurement Matrix');

y = phi*x;

subplot(2,2,3);
imagesc(reshape(y,[7 7]));
colormap gray
title(sprintf('Measurements (%d elements)',nMeasurements));

funObj = @(f)SquaredError(f,phi,y);
f = zeros(nVars,1);
options = [];
for lambda = 10.^[1:-1:-2]
    f = L1General2_PSSgb(funObj,f,lambda*ones(nVars,1),options);
    residual = norm(phi*f-y)
    reconstructionError = norm(f-x)

    subplot(2,2,4);
    imagesc(reshape(f,[16 16]));
colormap gray
    title(sprintf('Reconstructed (lambda = %.2f, error = %.2f)',lambda,reconstructionError));
    pause;
end
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 3.35502e-05 2.49414e+02 1.21044e+02    256
     2      3 1.00000e+00 7.50875e+01 1.22691e+01    256
     3      4 1.00000e+00 7.45935e+01 1.20244e+01    256
Backtracking...
     4      6 5.58838e-02 7.18038e+01 1.13193e+01    256
Backtracking...
     5      8 3.90337e-02 6.98779e+01 1.07825e+01    254
Backtracking...
     6     10 3.18286e-02 6.85003e+01 9.87135e+00    248
Backtracking...
     7     12 3.66454e-02 6.72506e+01 9.01928e+00    243
Backtracking...
     8     14 3.35227e-02 6.62559e+01 8.05011e+00    234
Backtracking...
     9     16 5.31147e-02 6.52563e+01 7.07645e+00    224
Backtracking...
    10     18 9.72282e-02 6.42318e+01 8.19072e+00    215
Backtracking...
    11     20 1.44341e-01 6.33824e+01 9.56689e+00    200
Backtracking...
    12     22 3.36791e-01 6.28361e+01 1.21604e+01    181
    13     23 1.00000e+00 6.18708e+01 1.13793e+01    171
Backtracking...
    14     25 2.96432e-02 6.13845e+01 1.14261e+01    165
Backtracking...
    15     27 4.21529e-02 6.08119e+01 1.13936e+01    156
Backtracking...
    16     29 4.63575e-02 6.02913e+01 1.13218e+01    146
Backtracking...
    17     31 6.34011e-02 5.96866e+01 1.07890e+01    137
Backtracking...
    18     33 7.40129e-02 5.88998e+01 1.01510e+01    128
Backtracking...
    19     35 1.75350e-01 5.77169e+01 1.01594e+01    112
    20     36 1.00000e+00 5.58082e+01 1.38145e+01     64
Backtracking...
    21     38 4.43738e-01 5.40059e+01 1.16370e+01     44
Backtracking...
    22     40 3.46341e-01 5.31614e+01 5.14224e+00    253
Backtracking...
    23     42 1.82283e-01 5.29936e+01 4.57903e+00    200
Backtracking...
    24     44 3.57707e-01 5.28811e+01 5.57518e+00    147
    25     45 1.00000e+00 5.28628e+01 7.59500e+00     90
    26     46 1.00000e+00 5.23126e+01 2.68277e+00     70
    27     47 1.00000e+00 5.21133e+01 3.69019e+00     53
    28     48 1.00000e+00 5.20151e+01 4.82135e+00     41
    29     49 1.00000e+00 5.17088e+01 3.80668e+00     37
Backtracking...
    30     51 2.63284e-01 5.13077e+01 4.90943e+00     32
    31     52 1.00000e+00 5.08179e+01 5.47259e+00     25
    32     53 1.00000e+00 5.02276e+01 4.54655e+00     21
Backtracking...
    33     55 2.74455e-02 5.00975e+01 4.24341e+00    166
Backtracking...
    34     57 4.68518e-02 4.99615e+01 3.75771e+00     83
Backtracking...
    35     59 2.49336e-01 4.95742e+01 2.71343e+00     98
Backtracking...
    36     61 5.12947e-02 4.95548e+01 3.20811e+00     47
Backtracking...
    37     63 4.22201e-03 4.95288e+01 3.15771e+00     46
Backtracking...
    38     65 2.25267e-01 4.93976e+01 3.95104e+00     38
    39     66 1.00000e+00 4.93048e+01 6.97752e+00     20
    40     67 1.00000e+00 4.90130e+01 4.07106e+00     19
Backtracking...
    41     69 2.96373e-01 4.89290e+01 2.51626e+00    155
Backtracking...
    42     71 1.77165e-02 4.89241e+01 2.65326e+00    144
Backtracking...
    43     73 8.54997e-02 4.89215e+01 2.88308e+00    132
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
    44     79 5.88547e-04 4.89215e+01 2.88672e+00    131
Backtracking...
    45     81 4.24407e-01 4.88942e+01 3.00351e+00     81
Backtracking...
    46     83 2.89523e-01 4.88677e+01 3.54518e+00     68
Backtracking...
    47     85 1.96644e-01 4.88420e+01 3.66921e+00     54
Backtracking...
    48     87 3.08337e-01 4.88323e+01 3.99057e+00     46
    49     88 1.00000e+00 4.86986e+01 3.53232e+00     26
Backtracking...
    50     90 5.60421e-01 4.85536e+01 3.97612e+00     21
Backtracking...
    51     92 1.69999e-01 4.85064e+01 2.92795e+00     20
    52     93 1.00000e+00 4.84441e+01 2.74973e+00     20
Backtracking...
    53     95 2.33193e-01 4.82388e+01 3.14110e+00     22
Backtracking...
    54     97 5.33995e-01 4.81137e+01 3.96201e+00     16
Backtracking...
    55     99 1.10694e-02 4.80970e+01 3.84399e+00     15
Backtracking...
    56    101 3.54207e-02 4.80793e+01 4.02648e+00     15
Backtracking...
    57    103 2.79191e-01 4.80710e+01 3.57274e+00     13
    58    104 1.00000e+00 4.80036e+01 1.39523e+00     13
    59    105 1.00000e+00 4.79970e+01 2.39056e+00     31
Backtracking...
    60    107 2.61682e-01 4.79932e+01 2.41536e+00     26
Backtracking...
    61    109 3.66070e-01 4.79906e+01 2.22905e+00     24
    62    110 1.00000e+00 4.79802e+01 2.18098e+00     22
    63    111 1.00000e+00 4.79649e+01 2.52345e+00     17
    64    112 1.00000e+00 4.79286e+01 2.00820e+00     15
Backtracking...
    65    114 1.00000e-03 4.79280e+01 2.00336e+00     15
Backtracking...
    66    116 4.84845e-01 4.78797e+01 1.79993e+00     12
Backtracking...
Backtracking...
    67    119 8.01789e-02 4.78755e+01 1.47826e+00     12
    68    120 1.00000e+00 4.78534e+01 1.70749e+00     13
Backtracking...
    69    122 2.16452e-01 4.78455e+01 1.38289e+00     12
    70    123 1.00000e+00 4.78335e+01 2.19775e+00     18
Backtracking...
    71    125 1.21312e-02 4.78286e+01 2.18760e+00     17
    72    126 1.00000e+00 4.78151e+01 1.58356e+00     16
Backtracking...
    73    128 4.16636e-01 4.78139e+01 2.62897e-01     17
    74    129 1.00000e+00 4.78139e+01 4.58304e-01     15
    75    130 1.00000e+00 4.78085e+01 2.25101e-01     15
    76    131 1.00000e+00 4.78077e+01 2.89047e-01     17
    77    132 1.00000e+00 4.78050e+01 1.02011e-01     16
    78    133 1.00000e+00 4.78032e+01 1.17981e-01     16
Backtracking...
    79    135 3.88570e-01 4.78023e+01 1.10667e-01     15
Backtracking...
    80    137 3.96675e-01 4.78014e+01 2.01171e-01     15
Backtracking...
    81    139 4.24047e-01 4.78003e+01 6.48385e-02     15
    82    140 1.00000e+00 4.77999e+01 4.88097e-02     15
    83    141 1.00000e+00 4.77998e+01 6.40268e-02     15
    84    142 1.00000e+00 4.77991e+01 4.83560e-02     15
Backtracking...
    85    144 3.71882e-01 4.77990e+01 3.98244e-02     15
    86    145 1.00000e+00 4.77988e+01 2.93461e-02     15
    87    146 1.00000e+00 4.77987e+01 2.24206e-02     15
    88    147 1.00000e+00 4.77987e+01 1.82457e-02     15
    89    148 1.00000e+00 4.77987e+01 1.31130e-02     15
    90    149 1.00000e+00 4.77987e+01 1.45262e-02     15
    91    150 1.00000e+00 4.77987e+01 1.41998e-02     15
    92    151 1.00000e+00 4.77987e+01 1.17305e-02     15
    93    152 1.00000e+00 4.77987e+01 7.98338e-03     15
    94    153 1.00000e+00 4.77987e+01 5.00422e-03     15
    95    154 1.00000e+00 4.77987e+01 3.40224e-03     15
    96    155 1.00000e+00 4.77987e+01 1.98160e-03     15
    97    156 1.00000e+00 4.77987e+01 1.05096e-03     15
    98    157 1.00000e+00 4.77987e+01 5.64512e-04     15
    99    158 1.00000e+00 4.77987e+01 3.50266e-04     15
   100    159 1.00000e+00 4.77987e+01 1.75665e-04     15
   101    160 1.00000e+00 4.77987e+01 7.10882e-05     15
   102    161 1.00000e+00 4.77987e+01 1.34542e-05     15
Directional derivative below progTol

residual =

    1.4837


reconstructionError =

    0.5158

  Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
     1      3 1.60236e-04 5.68849e+00 2.34637e+00    256
     2      4 1.00000e+00 5.67463e+00 2.32625e+00    256
Backtracking...
     3      6 2.25231e-02 5.63884e+00 2.27424e+00    256
Backtracking...
     4      8 1.82919e-02 5.61157e+00 2.19021e+00    243
Backtracking...
     5     10 2.50015e-02 5.59090e+00 2.02519e+00    230
Backtracking...
     6     12 6.56450e-02 5.56967e+00 1.79559e+00    210
Backtracking...
     7     14 1.50409e-01 5.55205e+00 1.43738e+00    185
     8     15 1.00000e+00 5.53844e+00 2.08636e+00    155
     9     16 1.00000e+00 5.51437e+00 1.35597e+00    213
    10     17 1.00000e+00 5.49252e+00 1.41764e+00    176
Backtracking...
    11     19 3.68872e-02 5.48114e+00 1.33078e+00    161
Backtracking...
    12     21 1.06121e-01 5.46949e+00 1.11425e+00    147
Backtracking...
    13     23 2.31599e-01 5.45976e+00 1.30507e+00    128
    14     24 1.00000e+00 5.44864e+00 1.69869e+00    108
    15     25 1.00000e+00 5.42534e+00 1.81223e+00    104
Backtracking...
    16     27 2.64930e-01 5.40175e+00 1.95337e+00    103
    17     28 1.00000e+00 5.36302e+00 1.93737e+00     98
    18     29 1.00000e+00 5.30890e+00 1.12735e+00     84
Backtracking...
    19     31 1.48389e-01 5.29321e+00 9.68236e-01     73
    20     32 1.00000e+00 5.26064e+00 9.60619e-01     57
Backtracking...
Backtracking...
    21     35 1.84666e-02 5.25926e+00 1.51982e+00     55
    22     36 1.00000e+00 5.23156e+00 1.45435e+00     46
Backtracking...
Backtracking...
    23     39 6.37792e-02 5.19768e+00 2.08783e+00     27
Backtracking...
    24     41 2.63467e-01 5.19192e+00 1.53348e+00    230
    25     42 1.00000e+00 5.18808e+00 1.36498e+00    171
    26     43 1.00000e+00 5.18357e+00 1.25639e+00    126
    27     44 1.00000e+00 5.18325e+00 1.53692e+00     92
    28     45 1.00000e+00 5.16997e+00 1.00934e+00     87
    29     46 1.00000e+00 5.16124e+00 1.25961e+00     72
    30     47 1.00000e+00 5.13198e+00 1.09025e+00     37
Backtracking...
    31     49 4.13547e-02 5.13057e+00 1.26071e+00     35
    32     50 1.00000e+00 5.11300e+00 1.17742e+00     29
Backtracking...
    33     52 2.80916e-01 5.07539e+00 1.84497e+00     21
Backtracking...
    34     54 5.23250e-02 5.07332e+00 1.82016e+00     25
Backtracking...
    35     56 1.46962e-02 5.04210e+00 9.63966e-01     19
    36     57 1.00000e+00 5.02598e+00 4.11094e-01     17
    37     58 1.00000e+00 5.02203e+00 6.04572e-01     14
Backtracking...
    38     60 3.86740e-01 5.01287e+00 8.12598e-01     14
    39     61 1.00000e+00 5.01036e+00 1.07048e+00     10
Backtracking...
    40     63 2.48744e-01 4.99471e+00 3.52853e-01     10
    41     64 1.00000e+00 4.99341e+00 8.89207e-01     95
Backtracking...
    42     66 2.54049e-01 4.99044e+00 7.83691e-01     71
Backtracking...
    43     68 1.98328e-01 4.98749e+00 4.74051e-01     65
Backtracking...
Backtracking...
    44     71 8.60715e-02 4.98729e+00 5.06820e-01     56
    45     72 1.00000e+00 4.98350e+00 3.42037e-01     22
Backtracking...
Backtracking...
    46     75 4.72320e-02 4.98338e+00 3.28807e-01     18
Backtracking...
Backtracking...
    47     78 7.07891e-02 4.98323e+00 2.25100e-01     17
Backtracking...
    48     80 4.18919e-01 4.98293e+00 1.94093e-01     13
    49     81 1.00000e+00 4.98238e+00 1.17359e-01     11
    50     82 1.00000e+00 4.98198e+00 1.29208e-01     13
    51     83 1.00000e+00 4.98051e+00 1.25856e-01     10
    52     84 1.00000e+00 4.97900e+00 1.29891e-01      9
    53     85 1.00000e+00 4.97872e+00 1.88020e-01      8
Backtracking...
    54     87 4.30720e-03 4.97869e+00 1.87528e-01      8
    55     88 1.00000e+00 4.97835e+00 4.69479e-02      8
    56     89 1.00000e+00 4.97827e+00 3.12822e-02      8
    57     90 1.00000e+00 4.97821e+00 2.65621e-02     13
    58     91 1.00000e+00 4.97819e+00 6.62817e-02     14
Backtracking...
Backtracking...
    59     94 5.17236e-04 4.97818e+00 6.93045e-02     14
Backtracking...
    60     96 3.43483e-01 4.97814e+00 4.80769e-02     13
    61     97 1.00000e+00 4.97811e+00 2.36317e-02     12
    62     98 1.00000e+00 4.97811e+00 4.64090e-02     15
Backtracking...
Backtracking...
    63    101 5.21036e-03 4.97811e+00 4.71775e-02     13
Backtracking...
Backtracking...
    64    104 4.48009e-04 4.97811e+00 4.74027e-02     13
Backtracking...
    65    106 8.98018e-02 4.97810e+00 3.76276e-02     13
Backtracking...
    66    108 2.87381e-02 4.97808e+00 3.60121e-02     12
    67    109 1.00000e+00 4.97804e+00 2.25433e-02     12
    68    110 1.00000e+00 4.97799e+00 6.35032e-03     13
Backtracking...
    69    112 1.72141e-01 4.97799e+00 6.71866e-03     12
    70    113 1.00000e+00 4.97799e+00 5.32645e-03     15
Backtracking...
    71    115 8.01806e-02 4.97799e+00 6.22308e-03     14
Backtracking...
    72    117 3.96239e-01 4.97799e+00 9.08961e-03     14
    73    118 1.00000e+00 4.97799e+00 5.18465e-03     14
    74    119 1.00000e+00 4.97799e+00 4.70484e-03     15
    75    120 1.00000e+00 4.97799e+00 1.20413e-02     15
Backtracking...
    76    122 4.91569e-01 4.97799e+00 5.21229e-03     15
    77    123 1.00000e+00 4.97799e+00 1.72499e-03     15
    78    124 1.00000e+00 4.97799e+00 1.38505e-03     15
    79    125 1.00000e+00 4.97799e+00 5.78246e-04     15
    80    126 1.00000e+00 4.97799e+00 2.71256e-04     15
    81    127 1.00000e+00 4.97799e+00 2.37731e-04     15
    82    128 1.00000e+00 4.97799e+00 1.53940e-04     15
    83    129 1.00000e+00 4.97799e+00 4.19501e-05     15
Directional derivative below progTol

residual =

    0.1484


reconstructionError =

    0.0516

  Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
     1      3 1.60236e-04 5.06885e-01 2.34663e-01    256
     2      4 1.00000e+00 5.06747e-01 2.32651e-01    256
Backtracking...
     3      6 2.25218e-02 5.06389e-01 2.27449e-01    256
Backtracking...
     4      8 1.82910e-02 5.06116e-01 2.19045e-01    243
Backtracking...
     5     10 2.50012e-02 5.05909e-01 2.02543e-01    230
Backtracking...
     6     12 6.56438e-02 5.05697e-01 1.79583e-01    210
Backtracking...
     7     14 1.50413e-01 5.05521e-01 1.43745e-01    185
     8     15 1.00000e+00 5.05385e-01 2.08634e-01    155
     9     16 1.00000e+00 5.05144e-01 1.35624e-01    213
    10     17 1.00000e+00 5.04926e-01 1.41787e-01    176
Backtracking...
    11     19 3.68734e-02 5.04812e-01 1.33108e-01    161
Backtracking...
    12     21 1.06018e-01 5.04695e-01 1.11461e-01    147
Backtracking...
    13     23 2.31630e-01 5.04598e-01 1.30482e-01    128
    14     24 1.00000e+00 5.04487e-01 1.69865e-01    108
    15     25 1.00000e+00 5.04254e-01 1.81175e-01    104
Backtracking...
    16     27 2.64380e-01 5.04018e-01 1.95298e-01    103
    17     28 1.00000e+00 5.03631e-01 1.93785e-01     98
    18     29 1.00000e+00 5.03090e-01 1.12675e-01     84
Backtracking...
    19     31 1.49860e-01 5.02932e-01 9.69812e-02     73
    20     32 1.00000e+00 5.02606e-01 9.59290e-02     57
Backtracking...
Backtracking...
    21     35 1.81477e-02 5.02589e-01 1.48497e-01     55
    22     36 1.00000e+00 5.02312e-01 1.45984e-01     46
Backtracking...
Backtracking...
    23     39 8.50074e-02 5.02006e-01 2.19799e-01     27
Backtracking...
    24     41 2.42453e-01 5.01938e-01 1.58443e-01    248
    25     42 1.00000e+00 5.01897e-01 1.27765e-01    172
    26     43 1.00000e+00 5.01855e-01 1.18496e-01    125
Backtracking...
    27     45 3.09388e-01 5.01819e-01 1.06252e-01     97
    28     46 1.00000e+00 5.01758e-01 1.13786e-01     72
Backtracking...
    29     48 3.14004e-01 5.01709e-01 1.49059e-01     43
    30     49 1.00000e+00 5.01571e-01 1.05665e-01     40
    31     50 1.00000e+00 5.01059e-01 2.32989e-01     24
Backtracking...
    32     52 1.55235e-01 5.00981e-01 1.55188e-01    155
Backtracking...
    33     54 1.30240e-01 5.00876e-01 1.12352e-01    128
Backtracking...
    34     56 6.52915e-03 5.00872e-01 1.12149e-01    114
Backtracking...
    35     58 4.02910e-01 5.00750e-01 1.57582e-01     45
Backtracking...
    36     60 2.11957e-03 5.00747e-01 1.55973e-01     44
Backtracking...
    37     62 1.95066e-01 5.00675e-01 1.38185e-01     26
Backtracking...
    38     64 1.91051e-03 5.00664e-01 1.40163e-01     26
Backtracking...
    39     66 1.42964e-01 5.00432e-01 1.63985e-01     23
Backtracking...
Backtracking...
    40     69 1.39894e-01 5.00275e-01 6.71169e-02     18
Backtracking...
Backtracking...
    41     72 1.55481e-04 5.00259e-01 5.90533e-02     78
Backtracking...
Backtracking...
    42     75 1.77622e-05 5.00243e-01 4.83415e-02     49
Backtracking...
    43     77 3.47796e-02 5.00232e-01 3.59950e-02     37
Backtracking...
    44     79 4.07696e-01 5.00154e-01 4.49068e-02     30
Backtracking...
    45     81 1.00000e-03 5.00146e-01 4.15653e-02     30
    46     82 1.00000e+00 5.00122e-01 8.86935e-02     21
Backtracking...
Backtracking...
    47     85 1.06394e-06 5.00119e-01 8.87844e-02     21
Backtracking...
    48     87 2.58230e-01 5.00042e-01 7.43836e-02     19
Backtracking...
Backtracking...
    49     90 3.44717e-04 5.00039e-01 7.22945e-02     19
Backtracking...
    50     92 4.86071e-01 4.99980e-01 5.27179e-02     18
Backtracking...
    51     94 1.75207e-01 4.99958e-01 4.10484e-02     17
Backtracking...
Backtracking...
    52     97 2.51529e-04 4.99956e-01 3.83362e-02     17
Backtracking...
    53     99 2.99588e-01 4.99923e-01 4.59696e-02     16
    54    100 1.00000e+00 4.99882e-01 3.31557e-02     13
Backtracking...
Backtracking...
    55    103 8.47893e-05 4.99879e-01 3.47154e-02     13
Backtracking...
    56    105 3.64022e-01 4.99860e-01 2.26680e-02     13
    57    106 1.00000e+00 4.99822e-01 4.35095e-02     43
Backtracking...
    58    108 8.33157e-03 4.99821e-01 4.41126e-02     39
Backtracking...
Backtracking...
    59    111 1.78770e-03 4.99821e-01 4.40774e-02     38
Backtracking...
Backtracking...
    60    114 1.63657e-02 4.99820e-01 4.33328e-02     33
Backtracking...
    61    116 1.00000e-03 4.99819e-01 4.19340e-02     30
Backtracking...
    62    118 2.45278e-03 4.99818e-01 4.15687e-02     29
Backtracking...
    63    120 2.68086e-01 4.99817e-01 4.30702e-02     23
Backtracking...
    64    122 4.58076e-03 4.99816e-01 4.25081e-02     22
    65    123 1.00000e+00 4.99808e-01 1.02988e-02     14
Backtracking...
    66    125 1.00000e-03 4.99808e-01 1.08631e-02     23
Backtracking...
Backtracking...
    67    128 2.78058e-05 4.99807e-01 1.34438e-02     21
Backtracking...
    68    130 5.91947e-02 4.99806e-01 1.70758e-02     16
Backtracking...
Backtracking...
Backtracking...
    69    134 1.12997e-05 4.99806e-01 1.75139e-02     14
Backtracking...
Backtracking...
Backtracking...
Backtracking...
    70    139 6.92373e-08 4.99806e-01 9.47738e-03     11
Backtracking...
Backtracking...
Backtracking...
    71    143 6.71580e-08 4.99805e-01 1.01570e-02     17
Backtracking...
Backtracking...
    72    146 1.03015e-05 4.99805e-01 1.09567e-02     14
Backtracking...
Backtracking...
    73    149 1.27424e-06 4.99805e-01 1.09161e-02     16
Backtracking...
Backtracking...
    74    152 4.20603e-04 4.99805e-01 1.29725e-02     14
Backtracking...
    75    154 2.71789e-01 4.99801e-01 1.65521e-02     12
Backtracking...
    76    156 3.59669e-01 4.99790e-01 1.49572e-02     11
Backtracking...
    77    158 1.00000e-03 4.99790e-01 1.46746e-02     11
Backtracking...
    78    160 2.04111e-01 4.99788e-01 1.18333e-02     11
Backtracking...
    79    162 4.47321e-01 4.99784e-01 5.64353e-03     11
    80    163 1.00000e+00 4.99782e-01 7.52085e-03     12
Backtracking...
    81    165 1.17572e-02 4.99782e-01 6.50864e-03     20
Backtracking...
    82    167 1.81064e-03 4.99782e-01 5.86296e-03     15
    83    168 1.00000e+00 4.99781e-01 3.70905e-03     17
Backtracking...
    84    170 1.00000e-03 4.99781e-01 3.10069e-03     16
Backtracking...
    85    172 6.34585e-02 4.99781e-01 1.41734e-03     14
    86    173 1.00000e+00 4.99780e-01 2.46138e-03     14
Backtracking...
Backtracking...
    87    176 2.12295e-03 4.99780e-01 1.41918e-03     14
    88    177 1.00000e+00 4.99780e-01 1.25534e-03     14
    89    178 1.00000e+00 4.99780e-01 3.82409e-03     15
Backtracking...
    90    180 3.79342e-02 4.99780e-01 3.33626e-03     15
Backtracking...
    91    182 5.00220e-03 4.99780e-01 3.38568e-03     15
Backtracking...
    92    184 3.86838e-01 4.99780e-01 2.71508e-03     15
    93    185 1.00000e+00 4.99780e-01 1.49377e-03     17
Backtracking...
Backtracking...
    94    188 4.42474e-02 4.99780e-01 1.45697e-03     16
Backtracking...
    95    190 5.95538e-03 4.99780e-01 1.48054e-03     16
    96    191 1.00000e+00 4.99780e-01 1.78872e-03     15
    97    192 1.00000e+00 4.99780e-01 6.27976e-04     15
    98    193 1.00000e+00 4.99780e-01 8.24886e-04     15
    99    194 1.00000e+00 4.99780e-01 6.87867e-04     15
   100    195 1.00000e+00 4.99780e-01 3.09047e-04     15
   101    196 1.00000e+00 4.99780e-01 1.95504e-04     15
   102    197 1.00000e+00 4.99780e-01 2.07888e-04     15
   103    198 1.00000e+00 4.99780e-01 1.90420e-04     15
   104    199 1.00000e+00 4.99780e-01 1.05667e-04     15
   105    200 1.00000e+00 4.99780e-01 6.84437e-05     15
Directional derivative below progTol

residual =

    0.0148


reconstructionError =

    0.0052

  Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
     1      3 1.60237e-04 5.00688e-02 2.34886e-02    256
     2      4 1.00000e+00 5.00675e-02 2.32872e-02    256
Backtracking...
     3      6 2.25313e-02 5.00639e-02 2.27664e-02    256
Backtracking...
     4      8 1.82917e-02 5.00612e-02 2.19247e-02    243
Backtracking...
     5     10 2.50170e-02 5.00591e-02 2.02739e-02    230
Backtracking...
     6     12 6.56545e-02 5.00570e-02 1.79802e-02    210
Backtracking...
     7     14 1.50108e-01 5.00552e-02 1.43878e-02    185
     8     15 1.00000e+00 5.00538e-02 2.08023e-02    155
     9     16 1.00000e+00 5.00514e-02 1.35429e-02    213
    10     17 1.00000e+00 5.00492e-02 1.42249e-02    176
Backtracking...
    11     19 3.60938e-02 5.00481e-02 1.33511e-02    161
Backtracking...
    12     21 1.05507e-01 5.00470e-02 1.12078e-02    146
Backtracking...
    13     23 2.23302e-01 5.00460e-02 1.28397e-02    128
    14     24 1.00000e+00 5.00449e-02 1.68841e-02    109
    15     25 1.00000e+00 5.00424e-02 1.71820e-02    105
Backtracking...
    16     27 1.76697e-01 5.00400e-02 1.79058e-02     94
    17     28 1.00000e+00 5.00380e-02 2.48689e-02     77
Backtracking...
    18     30 4.05591e-01 5.00354e-02 1.15970e-02    204
    19     31 1.00000e+00 5.00318e-02 1.51273e-02    112
Backtracking...
    20     33 2.12182e-01 5.00312e-02 2.04940e-02     70
    21     34 1.00000e+00 5.00287e-02 1.81159e-02     92
Backtracking...
    22     36 3.66422e-01 5.00237e-02 1.19228e-02     42
    23     37 1.00000e+00 5.00200e-02 1.10827e-02     31
    24     38 1.00000e+00 5.00160e-02 1.29873e-02     22
Backtracking...
Backtracking...
    25     41 2.57354e-01 5.00155e-02 2.46521e-02     20
Backtracking...
    26     43 3.44045e-01 5.00133e-02 1.51349e-02    254
    27     44 1.00000e+00 5.00123e-02 1.12316e-02    197
    28     45 1.00000e+00 5.00113e-02 1.30541e-02    134
    29     46 1.00000e+00 5.00106e-02 1.60556e-02     80
Backtracking...
    30     48 3.14852e-01 5.00095e-02 1.23390e-02     67
    31     49 1.00000e+00 5.00089e-02 1.24535e-02     66
    32     50 1.00000e+00 5.00088e-02 1.48105e-02     46
    33     51 1.00000e+00 5.00065e-02 7.97435e-03     42
Backtracking...
    34     53 3.30928e-01 5.00061e-02 8.89595e-03     35
Backtracking...
    35     55 1.86199e-01 5.00059e-02 8.23941e-03     32
Backtracking...
Backtracking...
    36     58 1.00000e-06 5.00057e-02 6.43879e-03     32
    37     59 1.00000e+00 5.00040e-02 6.64221e-03     29
Backtracking...
    38     61 4.64992e-03 5.00039e-02 6.50540e-03     29
Backtracking...
    39     63 2.89876e-01 5.00031e-02 8.43046e-03     26
Backtracking...
    40     65 1.52717e-02 5.00025e-02 7.55624e-03     25
    41     66 1.00000e+00 5.00008e-02 6.59390e-03     22
    42     67 1.00000e+00 5.00003e-02 4.99323e-03     21
Backtracking...
    43     69 4.18505e-01 5.00000e-02 3.26197e-03    150
Backtracking...
    44     71 1.81638e-01 5.00000e-02 4.00789e-03    126
Backtracking...
    45     73 4.36390e-03 5.00000e-02 3.96882e-03    124
Backtracking...
Backtracking...
    46     76 3.26264e-02 5.00000e-02 4.04217e-03    115
Backtracking...
    47     78 2.14299e-03 5.00000e-02 4.03723e-03    107
Backtracking...
    48     80 1.00000e-03 5.00000e-02 3.97649e-03    106
Backtracking...
    49     82 1.00000e-03 4.99999e-02 3.87416e-03    104
Backtracking...
    50     84 3.92451e-03 4.99999e-02 3.48224e-03    102
Backtracking...
    51     86 1.13284e-01 4.99999e-02 3.83733e-03     84
Backtracking...
    52     88 2.82300e-02 4.99999e-02 3.65085e-03     81
    53     89 1.00000e+00 4.99997e-02 2.53542e-03     66
Backtracking...
    54     91 2.46015e-01 4.99996e-02 3.51781e-03     65
Backtracking...
Backtracking...
    55     94 3.50341e-04 4.99996e-02 3.59789e-03     58
Backtracking...
    56     96 1.00000e-03 4.99995e-02 2.92133e-03     51
Backtracking...
    57     98 9.23513e-03 4.99995e-02 2.95674e-03     49
Backtracking...
    58    100 1.00000e-03 4.99995e-02 3.04431e-03     44
    59    101 1.00000e+00 4.99991e-02 4.12260e-03     38
Backtracking...
Backtracking...
    60    104 5.23265e-06 4.99991e-02 4.06727e-03     37
Backtracking...
    61    106 1.00000e-03 4.99990e-02 3.61422e-03     37
    62    107 1.00000e+00 4.99989e-02 4.04238e-03     21
Backtracking...
    63    109 2.88973e-03 4.99989e-02 3.43913e-03     21
Backtracking...
    64    111 2.29268e-01 4.99987e-02 2.12131e-03     21
    65    112 1.00000e+00 4.99985e-02 2.67954e-03     28
Backtracking...
Backtracking...
Backtracking...
    66    116 4.15876e-06 4.99985e-02 2.63678e-03     27
Progress in parameters or objective below progTol

residual =

    0.0015


reconstructionError =

   6.2500e-04

Chain-structured conditional random field

load wordData.mat
lambda = 1;

% Initialize
[w,v_start,v_end,v] = crfChain_initWeights(nFeatures,nStates,'zeros');
featureStart = cumsum([1 nFeatures(1:end)]); % data structure which relates high-level 'features' to elements of w
sentences = crfChain_initSentences(y);
nSentences = size(sentences,1);
maxSentenceLength = 1+max(sentences(:,2)-sentences(:,1));

fprintf('Training chain-structured CRF\n');
wv = [w(:);v_start;v_end;v(:)];
funObj = @(wv)crfChain_loss(wv,X,y,nStates,nFeatures,featureStart,sentences);
wv = L1General2_PSSgb(funObj,wv,lambda*ones(size(wv)),options);

% Split up weights
[w,v_start,v_end,v] = crfChain_splitWeights(wv,featureStart,nStates);

% Measure error
trainErr = 0;
trainZ = 0;
yhat = zeros(size(y));
for s = 1:nSentences
    y_s = y(sentences(s,1):sentences(s,2));
    [nodePot,edgePot]=crfChain_makePotentials(X,w,v_start,v_end,v,nFeatures,featureStart,sentences,s);
    [nodeBel,edgeBel,logZ] = crfChain_infer(nodePot,edgePot);
    [junk yhat(sentences(s,1):sentences(s,2))] = max(nodeBel,[],2);
end
trainErrRate = sum(y~=yhat)/length(y)

figure;
imagesc([y yhat]);
colormap gray
title('True sequence (left), sequence of marginally most likely states (right)');
figure;
subplot(1,2,1);
imagesc(w);
title('CRF Feature Potentials');
subplot(1,2,2);
imagesc([v_start' 0;v v_end]);
title('CRF Transition Potentials');
pause;
Training chain-structured CRF
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 4.86086e-04 1.14143e+03 1.17753e+02     78
     2      3 1.00000e+00 6.63197e+02 3.76440e+01     70
     3      4 1.00000e+00 5.82276e+02 4.57577e+01     68
     4      5 1.00000e+00 5.25860e+02 1.59569e+01     77
     5      6 1.00000e+00 4.91382e+02 6.99251e+00     71
     6      7 1.00000e+00 4.80566e+02 4.91975e+00     75
     7      8 1.00000e+00 4.75227e+02 3.16558e+00     75
     8      9 1.00000e+00 4.70818e+02 4.92525e+00     73
     9     10 1.00000e+00 4.68909e+02 5.41885e+00     68
    10     11 1.00000e+00 4.66096e+02 2.13411e+00     74
    11     12 1.00000e+00 4.64156e+02 2.64028e+00     69
    12     13 1.00000e+00 4.62693e+02 3.33599e+00     65
    13     14 1.00000e+00 4.61455e+02 1.75989e+00     66
    14     15 1.00000e+00 4.60940e+02 1.66060e+00     66
    15     16 1.00000e+00 4.60451e+02 1.22923e+00     65
    16     17 1.00000e+00 4.59987e+02 1.48194e+00     66
    17     18 1.00000e+00 4.59660e+02 1.21257e+00     64
    18     19 1.00000e+00 4.59362e+02 2.13169e+00     61
    19     20 1.00000e+00 4.59175e+02 1.26938e+00     59
    20     21 1.00000e+00 4.59062e+02 6.69563e-01     60
    21     22 1.00000e+00 4.58984e+02 1.11080e+00     60
    22     23 1.00000e+00 4.58956e+02 5.26510e-01     59
    23     24 1.00000e+00 4.58917e+02 4.26663e-01     59
    24     25 1.00000e+00 4.58896e+02 2.76077e-01     58
    25     26 1.00000e+00 4.58883e+02 2.61146e-01     58
    26     27 1.00000e+00 4.58861e+02 2.19739e-01     56
    27     28 1.00000e+00 4.58849e+02 3.06288e-01     56
    28     29 1.00000e+00 4.58835e+02 2.35700e-01     58
    29     30 1.00000e+00 4.58829e+02 1.19030e-01     58
    30     31 1.00000e+00 4.58823e+02 8.89767e-02     58
    31     32 1.00000e+00 4.58819e+02 1.49742e-01     58
    32     33 1.00000e+00 4.58817e+02 5.09001e-02     57
    33     34 1.00000e+00 4.58816e+02 4.17467e-02     58
    34     35 1.00000e+00 4.58815e+02 2.90042e-02     59
    35     36 1.00000e+00 4.58814e+02 4.90847e-02     59
    36     37 1.00000e+00 4.58814e+02 2.03209e-02     59
    37     38 1.00000e+00 4.58814e+02 1.70537e-02     59
    38     39 1.00000e+00 4.58814e+02 1.05995e-02     59
    39     40 1.00000e+00 4.58814e+02 2.75948e-02     58
    40     41 1.00000e+00 4.58814e+02 6.81921e-03     58
    41     42 1.00000e+00 4.58814e+02 5.25640e-03     58
    42     43 1.00000e+00 4.58814e+02 7.96413e-03     58
    43     44 1.00000e+00 4.58813e+02 8.19545e-03     58
    44     45 1.00000e+00 4.58813e+02 9.21045e-03     57
    45     46 1.00000e+00 4.58813e+02 2.46348e-03     57
    46     47 1.00000e+00 4.58813e+02 2.04623e-03     57
    47     48 1.00000e+00 4.58813e+02 1.51764e-03     57
    48     49 1.00000e+00 4.58813e+02 1.90082e-03     57
    49     50 1.00000e+00 4.58813e+02 6.58426e-04     57
    50     51 1.00000e+00 4.58813e+02 4.29058e-04     57
    51     52 1.00000e+00 4.58813e+02 3.59842e-04     57
    52     53 1.00000e+00 4.58813e+02 2.59996e-04     57
    53     54 1.00000e+00 4.58813e+02 1.56046e-04     57
    54     55 1.00000e+00 4.58813e+02 1.83365e-04     57
    55     56 1.00000e+00 4.58813e+02 5.31323e-05     57
    56     57 1.00000e+00 4.58813e+02 3.68219e-05     57
    57     58 1.00000e+00 4.58813e+02 2.98633e-05     57
Directional derivative below progTol

trainErrRate =

    0.1030

Graphical Lasso

lambda = .05;

% Generate a sparse positive-definite precision matrix
nNodes = 10;
adj = triu(rand(nNodes) > .75,1);
adj = setdiag(adj+adj',1);
P = randn(nNodes).*adj;
P = (P+P')/2;
tau = 1;
X = P + tau*eye(nNodes);
while ~ispd(X)
    tau = tau*2;
    X = P + tau*eye(nNodes);
end
mu = randn(nNodes,1);

% Sample from the GGM
C = inv(X);
R = chol(C)';
X = zeros(nInstances,nNodes);
for i = 1:nInstances
    X(i,:) = (mu + R*randn(nNodes,1))';
end

% Center and Standardize
X = standardizeCols(X);

% Train Full GGM
sigma_emp = cov(X);
nonZero = find(ones(nNodes));
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Kfull = eye(nNodes);
fprintf('Fitting full Gaussian graphical model\n');
Kfull(nonZero) = minFunc(funObj,Kfull(nonZero),options);

% Train GGM w/ L1-regularization
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Ksparse = eye(nNodes);
fprintf('Fitting sparse Gaussian graphical model\n');
Ksparse(nonZero) = L1General2_PSSgb(funObj,Ksparse(nonZero),lambda*ones(nNodes*nNodes,1),options);

figure;
subplot(1,2,1);
imagesc(Kfull);
title('Estimated Full Precision Matrix');
subplot(1,2,2);
imagesc(Ksparse);
title('Estimated Sparse Precision Matrix');
pause;
Fitting full Gaussian graphical model
 Iteration   FunEvals     Step Length    Function Val        Opt Cond
         1          4     4.56509e-01     8.71032e+00     3.59715e-01
         2          5     1.00000e+00     8.36687e+00     2.68547e-01
         3          6     1.00000e+00     8.03081e+00     1.37907e-01
         4          7     1.00000e+00     7.89537e+00     1.20463e-01
         5          8     1.00000e+00     7.82045e+00     6.59930e-02
         6          9     1.00000e+00     7.79321e+00     4.23325e-02
         7         10     1.00000e+00     7.78270e+00     4.04436e-02
         8         11     1.00000e+00     7.77689e+00     2.48096e-02
         9         12     1.00000e+00     7.77590e+00     1.28531e-02
        10         13     1.00000e+00     7.77565e+00     7.40367e-03
        11         14     1.00000e+00     7.77539e+00     4.47854e-03
        12         15     1.00000e+00     7.77522e+00     5.34326e-03
        13         16     1.00000e+00     7.77496e+00     4.07508e-03
        14         17     1.00000e+00     7.77467e+00     2.88622e-03
        15         19     2.24018e-01     7.77463e+00     2.41829e-03
        16         20     1.00000e+00     7.77458e+00     6.65333e-04
        17         21     1.00000e+00     7.77458e+00     8.04254e-05
        18         22     1.00000e+00     7.77458e+00     1.78096e-05
        19         23     1.00000e+00     7.77458e+00     1.18396e-05
Directional Derivative below progTol
Fitting sparse Gaussian graphical model
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 9.68356e-02 1.02198e+01 5.51510e-01     68
Backtracking...
Halving Step Size
     2      4 5.00000e-01 9.47195e+00 2.65052e-01     68
     3      5 1.00000e+00 9.30345e+00 1.55103e-01     70
     4      6 1.00000e+00 9.14340e+00 9.95526e-02     66
     5      7 1.00000e+00 9.11583e+00 8.57312e-02     58
     6      8 1.00000e+00 9.09648e+00 3.34571e-02     62
     7      9 1.00000e+00 9.09328e+00 2.54082e-02     60
     8     10 1.00000e+00 9.09255e+00 1.75467e-02     62
     9     11 1.00000e+00 9.09176e+00 7.25425e-03     58
    10     12 1.00000e+00 9.09167e+00 4.48355e-03     60
    11     13 1.00000e+00 9.09164e+00 1.20526e-03     60
    12     14 1.00000e+00 9.09163e+00 1.31647e-03     58
    13     15 1.00000e+00 9.09162e+00 1.40570e-03     58
    14     16 1.00000e+00 9.09162e+00 6.74013e-04     58
    15     17 1.00000e+00 9.09162e+00 4.52143e-04     58
    16     18 1.00000e+00 9.09162e+00 1.52627e-04     58
    17     19 1.00000e+00 9.09162e+00 1.13633e-04     58
    18     20 1.00000e+00 9.09162e+00 1.06931e-04     58
    19     21 1.00000e+00 9.09162e+00 1.21023e-04     58
    20     22 1.00000e+00 9.09162e+00 2.43673e-05     58
    21     23 1.00000e+00 9.09162e+00 1.73472e-05     58
    22     24 1.00000e+00 9.09162e+00 9.34430e-06     58
First-order optimality below optTol

Markov Random Field Structure Learning

lambda = 5;

% Generate Data
nInstances = 500;
nNodes = 10;
edgeDensity = .5;
nStates = 2;
ising = 1;
tied = 0;
useMex = 1;
y = UGM_generate(nInstances,0,nNodes,edgeDensity,nStates,ising,tied);

% Set up MRF and initialize parameters
adjInit = fullAdjMatrix(nNodes);
edgeStruct = UGM_makeEdgeStruct(adjInit,nStates,useMex);
Xnode = ones(nInstances,1,nNodes);
Xedge = ones(nInstances,1,edgeStruct.nEdges);
[nodeMap,edgeMap] = UGM_makeMRFmaps(nNodes,edgeStruct,ising,tied);
nNodeParams = max(nodeMap(:));
nParams = max(edgeMap(:));
nEdgeParams = nParams-nNodeParams;
w = zeros(nParams,1);

% Make Edge Regularizer
regularizer = [zeros(nNodeParams,1);lambda*ones(nEdgeParams,1)];

% Inference Method
inferFunc = @UGM_Infer_Exact;

% Train
suffStat = UGM_MRF_computeSuffStat(y,nodeMap,edgeMap,edgeStruct);
funObj = @(w)UGM_MRF_NLL(w,nInstances,suffStat,nodeMap,edgeMap,edgeStruct,inferFunc);
w = L1General2_PSSgb(funObj,w,regularizer,options);

% Find Active Edges
adjFinal = zeros(nNodes);
for e = 1:edgeStruct.nEdges
	params = edgeMap(:,:,e);
	params = params(params(:)~=0);
    if any(w(params(:)) ~= 0)
        n1 = edgeStruct.edgeEnds(e,1);
        n2 = edgeStruct.edgeEnds(e,2);
        adjFinal(n1,n2) = 1;
        adjFinal(n2,n1) = 1;
    end
end

% Display Graph (requires graphviz)
figure;
drawGraph(adjFinal);
title('Estimated Structure');
pause
Exact Sampling
  Iter fEvals      stepLen         fVal      optCond    nnz
     1      2 9.66837e-05 3.28728e+03 2.28945e+02     64
Backtracking...
     2      4 3.33072e-01 1.73173e+03 1.66001e+02     64
     3      5 1.00000e+00 1.52885e+03 8.51719e+01     63
     4      6 1.00000e+00 1.47799e+03 9.43565e+01     62
     5      7 1.00000e+00 1.39067e+03 6.06855e+01     60
     6      8 1.00000e+00 1.32590e+03 3.55498e+01     62
     7      9 1.00000e+00 1.30238e+03 2.27209e+01     59
     8     10 1.00000e+00 1.29383e+03 1.91823e+01     63
     9     11 1.00000e+00 1.28507e+03 1.30392e+01     58
    10     12 1.00000e+00 1.27722e+03 1.15441e+01     63
    11     13 1.00000e+00 1.26751e+03 2.03941e+01     55
    12     14 1.00000e+00 1.26067e+03 9.82268e+00     60
    13     15 1.00000e+00 1.25943e+03 2.51742e+01     51
    14     16 1.00000e+00 1.25590e+03 1.67151e+01     60
    15     17 1.00000e+00 1.25460e+03 9.33454e+00     58
    16     18 1.00000e+00 1.25265e+03 6.48217e+00     55
    17     19 1.00000e+00 1.25062e+03 1.51427e+01     49
    18     20 1.00000e+00 1.24950e+03 5.93236e+00     57
    19     21 1.00000e+00 1.24880e+03 6.35659e+00     54
    20     22 1.00000e+00 1.24809e+03 7.18483e+00     49
    21     23 1.00000e+00 1.24596e+03 8.38261e+00     50
    22     24 1.00000e+00 1.24554e+03 7.47506e+00     48
    23     25 1.00000e+00 1.24423e+03 1.09210e+01     48
Backtracking...
    24     27 3.53224e-02 1.24412e+03 9.80552e+00     52
    25     28 1.00000e+00 1.24314e+03 4.80184e+00     47
    26     29 1.00000e+00 1.24306e+03 5.16322e+00     44
    27     30 1.00000e+00 1.24268e+03 3.45606e+00     44
    28     31 1.00000e+00 1.24220e+03 5.18534e+00     41
    29     32 1.00000e+00 1.24197e+03 5.47500e+00     41
    30     33 1.00000e+00 1.24148e+03 5.22893e+00     39
    31     34 1.00000e+00 1.24115e+03 4.99996e+00     40
    32     35 1.00000e+00 1.24104e+03 4.16699e+00     39
    33     36 1.00000e+00 1.24097e+03 3.30432e+00     39
    34     37 1.00000e+00 1.24089e+03 1.78895e+00     39
Backtracking...
Backtracking...
    35     40 4.80636e-02 1.24089e+03 9.29785e-01     38
    36     41 1.00000e+00 1.24088e+03 9.27248e-01     38
    37     42 1.00000e+00 1.24084e+03 7.14728e-01     38
    38     43 1.00000e+00 1.24080e+03 9.86125e-01     37
    39     44 1.00000e+00 1.24076e+03 1.40380e+00     37
    40     45 1.00000e+00 1.24072e+03 1.42731e+00     37
    41     46 1.00000e+00 1.24069e+03 1.03998e+00     36
Backtracking...
    42     48 5.10320e-03 1.24069e+03 7.79459e-01     36
    43     49 1.00000e+00 1.24068e+03 6.11590e-01     36
    44     50 1.00000e+00 1.24067e+03 1.02219e+00     37
    45     51 1.00000e+00 1.24066e+03 8.22553e-01     36
    46     52 1.00000e+00 1.24065e+03 5.66586e-01     36
    47     53 1.00000e+00 1.24064e+03 3.44650e-01     36
Backtracking...
    48     55 4.68473e-01 1.24064e+03 5.70245e-01     35
Backtracking...
    49     57 3.02884e-01 1.24063e+03 3.83324e-01     36
Backtracking...
    50     59 3.80878e-01 1.24063e+03 5.00663e-01     38
Backtracking...
    51     61 3.28994e-01 1.24063e+03 5.28505e-01     37
    52     62 1.00000e+00 1.24063e+03 1.72054e-01     37
    53     63 1.00000e+00 1.24063e+03 2.11417e-01     36
    54     64 1.00000e+00 1.24063e+03 4.02944e-01     35
    55     65 1.00000e+00 1.24063e+03 1.97025e-01     35
    56     66 1.00000e+00 1.24063e+03 1.31077e-01     35
    57     67 1.00000e+00 1.24062e+03 2.31428e-01     35
    58     68 1.00000e+00 1.24062e+03 1.78554e-01     35
    59     69 1.00000e+00 1.24062e+03 1.01245e-01     35
    60     70 1.00000e+00 1.24062e+03 9.32327e-02     35
    61     71 1.00000e+00 1.24062e+03 1.00376e-01     35
    62     72 1.00000e+00 1.24062e+03 8.95266e-02     35
    63     73 1.00000e+00 1.24062e+03 7.26738e-02     35
    64     74 1.00000e+00 1.24062e+03 4.67728e-02     35
    65     75 1.00000e+00 1.24062e+03 2.58079e-02     35
    66     76 1.00000e+00 1.24062e+03 2.28698e-02     35
    67     77 1.00000e+00 1.24062e+03 3.77576e-02     35
    68     78 1.00000e+00 1.24062e+03 3.47257e-02     35
    69     79 1.00000e+00 1.24062e+03 2.42335e-02     35
    70     80 1.00000e+00 1.24062e+03 1.67902e-02     35
    71     81 1.00000e+00 1.24062e+03 1.08630e-02     35
    72     82 1.00000e+00 1.24062e+03 8.51776e-03     35
    73     83 1.00000e+00 1.24062e+03 6.40910e-03     35
    74     84 1.00000e+00 1.24062e+03 4.00380e-03     35
    75     85 1.00000e+00 1.24062e+03 1.62359e-03     35
    76     86 1.00000e+00 1.24062e+03 6.93276e-04     35
    77     87 1.00000e+00 1.24062e+03 6.15525e-04     35
    78     88 1.00000e+00 1.24062e+03 5.46709e-04     35
    79     89 1.00000e+00 1.24062e+03 3.98170e-04     35
    80     90 1.00000e+00 1.24062e+03 2.63634e-04     35
    81     91 1.00000e+00 1.24062e+03 1.69251e-04     35
    82     92 1.00000e+00 1.24062e+03 1.40953e-04     35
    83     93 1.00000e+00 1.24062e+03 1.10144e-04     35
    84     94 1.00000e+00 1.24062e+03 4.16877e-05     35
Directional derivative below progTol
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)

Neural Network with Sparse Connections

fig = 100;
lambda = 1;
options.quadraticInit = 1; % Use quadratic initialization of line search
options.verbose = 0;

% Generate non-linear regression data set
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear',nInstances,nVars);

X = [ones(nInstances,1) X];
nVars = nVars+1;

% Train neural network w/ multiple hiden layers
nHidden = [9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
    nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);

funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
funObjL2 = @(w)penalizedL2(w,funObj,lambdaL2);
while 1
    w_old = wMLP;
    wMLP = L1General2_PSSgb(funObjL2,wMLP,lambda*ones(nParams,1),options);
    if norm(w_old-wMLP,inf) < 1e-4
        break;
	end
end

% Plot results
figure;
hold on
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Neural Net'});

% Form weights
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
    hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
    offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));

% Make adjacency matrix
adj = zeros(nVars+sum(nHidden)+1);
for i = 1:nVars
    for j = 1:nHidden(1)
        if abs(inputWeights(i,j)) > 1e-4
            adj(i,nVars+j) = 1;
        end
    end
end
for h = 1:length(nHidden)-1
    for i = 1:nHidden(h)
        for j = 1:nHidden(h+1)
            if abs(hiddenWeights{h}(i,j)) > 1e-4
                adj(nVars+sum(nHidden(1:h-1))+i,nVars+sum(nHidden(1:h))+j) = 1;
            end
        end
    end
end
for i = 1:nHidden(end)
    if abs(outputWeights(i)) > 1e-4
        adj(nVars+sum(nHidden(1:end-1))+i,end) = 1;
    end
end

labels = cell(length(adj),1);
for i = 1:nVars
    labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
    i = i + 1;
    labels{i,1} = sprintf('b_%d',h);
    for j = 2:nHidden(h)
        i = i + 1;
        labels{i,1} = sprintf('h_%d_%d',h,j);
    end
end
labels{end,1} = 'y';

% Plot Network
figure;
drawGraph(adj,'labels',labels);
title('Neural Network');
pause
Training neural network for regression...
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)

Deep Network with Sparse Connections

fig = 1000;
lambda = 2;
options.maxIter = 100; % Restart more often for non-convex
options.quadraticInit = 2; % Use quadratic initialization of line search
options.verbose = 0;

% Generate non-linear regression data set
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear2',nInstances,nVars);

X = [ones(nInstances,1) X];
nVars = nVars+1;

% Train neural network w/ multiple hiden layers
nHidden = [9 9 9 9 9 9 9 9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
    nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);

funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
funObjL2 = @(w)penalizedL2(w,funObj,lambdaL2);
for i = 1:250
    fprintf('Training:');
    w_old = wMLP;
    wMLP = L1General2_PSSgb(funObjL2,wMLP,lambda*ones(nParams,1),options);
    fprintf(' (nnz = %d, max change = %f)\n',nnz(wMLP),norm(w_old-wMLP,inf));
    if norm(w_old-wMLP,inf) < 1e-5
        break;
	end
end

% Plot results
figure;
hold on
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Deep Neural Net'});

% Form weights
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
    hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
    offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));

% Make adjacency matrix
adj = zeros(nVars+sum(nHidden)+1);
for i = 1:nVars
    for j = 1:nHidden(1)
        if abs(inputWeights(i,j)) > 1e-4
            adj(i,nVars+j) = 1;
        end
    end
end
for h = 1:length(nHidden)-1
    for i = 1:nHidden(h)
        for j = 1:nHidden(h+1)
            if abs(hiddenWeights{h}(i,j)) > 1e-4
                adj(nVars+sum(nHidden(1:h-1))+i,nVars+sum(nHidden(1:h))+j) = 1;
            end
        end
    end
end
for i = 1:nHidden(end)
    if abs(outputWeights(i)) > 1e-4
        adj(nVars+sum(nHidden(1:end-1))+i,end) = 1;
    end
end

labels = cell(length(adj),1);
for i = 1:nVars
    labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
    i = i + 1;
    labels{i,1} = sprintf('b_%d',h);
    for j = 2:nHidden(h)
        i = i + 1;
        labels{i,1} = sprintf('h_%d_%d',h,j);
    end
end
labels{end,1} = 'y';

% Plot Network
figure;
drawGraph(adj,'labels',labels);
title('Neural Network');
Training neural network for regression...
Training: (nnz = 667, max change = 0.967185)
Training: (nnz = 635, max change = 0.779682)
Training: (nnz = 641, max change = 2.636920)
Training: (nnz = 633, max change = 0.596286)
Training: (nnz = 595, max change = 0.402751)
Training: (nnz = 588, max change = 0.474491)
Training: (nnz = 566, max change = 0.237261)
Training: (nnz = 542, max change = 0.594388)
Training: (nnz = 475, max change = 0.411673)
Training: (nnz = 457, max change = 0.290917)
Training: (nnz = 404, max change = 0.177169)
Training: (nnz = 360, max change = 0.286962)
Training: (nnz = 351, max change = 0.245158)
Training: (nnz = 301, max change = 0.236724)
Training: (nnz = 253, max change = 0.197978)
Training: (nnz = 229, max change = 0.234248)
Training: (nnz = 214, max change = 0.110784)
Training: (nnz = 209, max change = 0.134172)
Training: (nnz = 200, max change = 0.112478)
Training: (nnz = 183, max change = 0.048467)
Training: (nnz = 183, max change = 0.075826)
Training: (nnz = 178, max change = 0.065721)
Training: (nnz = 175, max change = 0.090798)
Training: (nnz = 166, max change = 0.052506)
Training: (nnz = 165, max change = 0.098766)
Training: (nnz = 162, max change = 0.047602)
Training: (nnz = 152, max change = 0.220688)
Training: (nnz = 145, max change = 0.054457)
Training: (nnz = 150, max change = 0.055629)
Training: (nnz = 147, max change = 0.027674)
Training: (nnz = 142, max change = 0.021358)
Training: (nnz = 144, max change = 0.023483)
Training: (nnz = 135, max change = 0.016194)
Training: (nnz = 138, max change = 0.047708)
Training: (nnz = 144, max change = 0.363394)
Training: (nnz = 134, max change = 0.039826)
Training: (nnz = 135, max change = 0.031717)
Training: (nnz = 141, max change = 0.038360)
Training: (nnz = 131, max change = 0.017409)
Training: (nnz = 129, max change = 0.068170)
Training: (nnz = 117, max change = 0.039581)
Training: (nnz = 123, max change = 0.021510)
Training: (nnz = 109, max change = 0.033459)
Training: (nnz = 109, max change = 0.022375)
Training: (nnz = 110, max change = 0.018931)
Training: (nnz = 104, max change = 0.028466)
Training: (nnz = 116, max change = 0.054193)
Training: (nnz = 107, max change = 0.029349)
Training: (nnz = 106, max change = 0.016000)
Training: (nnz = 106, max change = 0.046723)
Training: (nnz = 104, max change = 0.038716)
Training: (nnz = 98, max change = 0.023524)
Training: (nnz = 98, max change = 0.023039)
Training: (nnz = 114, max change = 0.055210)
Training: (nnz = 99, max change = 0.036755)
Training: (nnz = 97, max change = 0.090063)
Training: (nnz = 88, max change = 0.045667)
Training: (nnz = 106, max change = 0.035059)
Training: (nnz = 102, max change = 0.076933)
Training: (nnz = 100, max change = 0.028952)
Training: (nnz = 92, max change = 0.051420)
Training: (nnz = 86, max change = 0.070376)
Training: (nnz = 83, max change = 0.036400)
Training: (nnz = 86, max change = 0.059172)
Training: (nnz = 77, max change = 0.032287)
Training: (nnz = 84, max change = 0.030322)
Training: (nnz = 80, max change = 0.026959)
Training: (nnz = 71, max change = 0.028908)
Training: (nnz = 75, max change = 0.020683)
Training: (nnz = 78, max change = 0.019452)
Training: (nnz = 78, max change = 0.016166)
Training: (nnz = 74, max change = 0.028601)
Training: (nnz = 68, max change = 0.021595)
Training: (nnz = 74, max change = 0.062828)
Training: (nnz = 73, max change = 0.030521)
Training: (nnz = 74, max change = 0.017069)
Training: (nnz = 69, max change = 0.011962)
Training: (nnz = 71, max change = 0.054461)
Training: (nnz = 72, max change = 0.018591)
Training: (nnz = 69, max change = 0.015757)
Training: (nnz = 70, max change = 0.016410)
Training: (nnz = 68, max change = 0.017009)
Training: (nnz = 62, max change = 0.009676)
Training: (nnz = 67, max change = 0.026478)
Training: (nnz = 60, max change = 0.012650)
Training: (nnz = 60, max change = 0.012972)
Training: (nnz = 62, max change = 0.025775)
Training: (nnz = 63, max change = 0.019782)
Training: (nnz = 57, max change = 0.003912)
Training: (nnz = 57, max change = 0.066164)
Training: (nnz = 56, max change = 0.017881)
Training: (nnz = 56, max change = 0.015960)
Training: (nnz = 49, max change = 0.003901)
Training: (nnz = 60, max change = 0.031464)
Training: (nnz = 52, max change = 0.007451)
Training: (nnz = 47, max change = 0.013513)
Training: (nnz = 48, max change = 0.018263)
Training: (nnz = 48, max change = 0.005887)
Training: (nnz = 49, max change = 0.011130)
Training: (nnz = 64, max change = 0.340443)
Training: (nnz = 57, max change = 0.026491)
Training: (nnz = 59, max change = 0.023941)
Training: (nnz = 56, max change = 0.012665)
Training: (nnz = 53, max change = 0.012904)
Training: (nnz = 48, max change = 0.002348)
Training: (nnz = 49, max change = 0.007044)
Training: (nnz = 46, max change = 0.002118)
Training: (nnz = 54, max change = 0.004446)
Training: (nnz = 43, max change = 0.001986)
Training: (nnz = 52, max change = 0.004125)
Training: (nnz = 42, max change = 0.002107)
Training: (nnz = 58, max change = 0.051137)
Training: (nnz = 49, max change = 0.004627)
Training: (nnz = 45, max change = 0.002313)
Training: (nnz = 41, max change = 0.001677)
Training: (nnz = 55, max change = 0.049206)
Training: (nnz = 52, max change = 0.005081)
Training: (nnz = 45, max change = 0.002384)
Training: (nnz = 54, max change = 0.015250)
Training: (nnz = 48, max change = 0.001602)
Training: (nnz = 48, max change = 0.005457)
Training: (nnz = 44, max change = 0.002047)
Training: (nnz = 57, max change = 0.070230)
Training: (nnz = 48, max change = 0.003937)
Training: (nnz = 54, max change = 0.017014)
Training: (nnz = 49, max change = 0.003003)
Training: (nnz = 47, max change = 0.002668)
Training: (nnz = 45, max change = 0.005518)
Training: (nnz = 46, max change = 0.003938)
Training: (nnz = 56, max change = 0.071594)
Training: (nnz = 54, max change = 0.004618)
Training: (nnz = 51, max change = 0.004196)
Training: (nnz = 47, max change = 0.002295)
Training: (nnz = 50, max change = 0.001950)
Training: (nnz = 49, max change = 0.029989)
Training: (nnz = 45, max change = 0.002197)
Training: (nnz = 42, max change = 0.004006)
Training: (nnz = 46, max change = 0.005113)
Training: (nnz = 44, max change = 0.002699)
Training: (nnz = 51, max change = 0.013256)
Training: (nnz = 47, max change = 0.002213)
Training: (nnz = 41, max change = 0.005064)
Training: (nnz = 40, max change = 0.001433)
Training: (nnz = 49, max change = 0.049151)
Training: (nnz = 45, max change = 0.002600)
Training: (nnz = 40, max change = 0.003848)
Training: (nnz = 44, max change = 0.005215)
Training: (nnz = 44, max change = 0.003579)
Training: (nnz = 42, max change = 0.010221)
Training: (nnz = 40, max change = 0.003269)
Training: (nnz = 43, max change = 0.002905)
Training: (nnz = 44, max change = 0.004377)
Training: (nnz = 39, max change = 0.003928)
Training: (nnz = 40, max change = 0.004021)
Training: (nnz = 47, max change = 0.004692)
Training: (nnz = 42, max change = 0.003924)
Training: (nnz = 39, max change = 0.003004)
Training: (nnz = 43, max change = 0.020384)
Training: (nnz = 38, max change = 0.002724)
Training: (nnz = 42, max change = 0.005549)
Training: (nnz = 39, max change = 0.003570)
Training: (nnz = 39, max change = 0.004197)
Training: (nnz = 36, max change = 0.002509)
Training: (nnz = 55, max change = 0.083566)
Training: (nnz = 54, max change = 0.004167)
Training: (nnz = 46, max change = 0.002941)
Training: (nnz = 43, max change = 0.011560)
Training: (nnz = 42, max change = 0.004004)
Training: (nnz = 39, max change = 0.005500)
Training: (nnz = 37, max change = 0.000033)
Training: (nnz = 41, max change = 0.004653)
Training: (nnz = 39, max change = 0.004093)
Training: (nnz = 36, max change = 0.003060)
Training: (nnz = 38, max change = 0.016099)
Training: (nnz = 37, max change = 0.003698)
Training: (nnz = 41, max change = 0.004591)
Training: (nnz = 38, max change = 0.003958)
Training: (nnz = 38, max change = 0.004611)
Training: (nnz = 52, max change = 0.012214)
Training: (nnz = 39, max change = 0.002305)
Training: (nnz = 40, max change = 0.003415)
Training: (nnz = 46, max change = 0.019619)
Training: (nnz = 39, max change = 0.002502)
Training: (nnz = 49, max change = 0.119394)
Training: (nnz = 52, max change = 0.002777)
Training: (nnz = 46, max change = 0.002635)
Training: (nnz = 40, max change = 0.001766)
Training: (nnz = 49, max change = 0.003978)
Training: (nnz = 39, max change = 0.003955)
Training: (nnz = 41, max change = 0.022689)
Training: (nnz = 44, max change = 0.003586)
Training: (nnz = 40, max change = 0.007178)
Training: (nnz = 43, max change = 0.011511)
Training: (nnz = 40, max change = 0.003482)
Training: (nnz = 38, max change = 0.006133)
Training: (nnz = 43, max change = 0.007837)
Training: (nnz = 36, max change = 0.004371)
Training: (nnz = 35, max change = 0.007612)
Training: (nnz = 62, max change = 0.778716)
Training: (nnz = 58, max change = 0.025350)
Training: (nnz = 47, max change = 0.007105)
Training: (nnz = 51, max change = 0.005689)
Training: (nnz = 57, max change = 0.003822)
Training: (nnz = 59, max change = 0.014219)
Training: (nnz = 49, max change = 0.001399)
Training: (nnz = 61, max change = 0.521767)
Training: (nnz = 58, max change = 0.027560)
Training: (nnz = 55, max change = 0.011300)
Training: (nnz = 55, max change = 0.013688)
Training: (nnz = 55, max change = 0.010813)
Training: (nnz = 53, max change = 0.013645)
Training: (nnz = 53, max change = 0.006686)
Training: (nnz = 58, max change = 0.018081)
Training: (nnz = 50, max change = 0.015506)
Training: (nnz = 51, max change = 0.002396)
Training: (nnz = 54, max change = 0.032205)
Training: (nnz = 49, max change = 0.006285)
Training: (nnz = 56, max change = 0.017473)
Training: (nnz = 47, max change = 0.003555)
Training: (nnz = 52, max change = 0.003757)
Training: (nnz = 59, max change = 0.029159)
Training: (nnz = 52, max change = 0.009495)
Training: (nnz = 51, max change = 0.008131)
Training: (nnz = 55, max change = 0.007305)
Training: (nnz = 54, max change = 0.007790)
Training: (nnz = 46, max change = 0.003035)
Training: (nnz = 45, max change = 0.007596)
Training: (nnz = 61, max change = 0.143070)
Training: (nnz = 60, max change = 0.025799)
Training: (nnz = 53, max change = 0.014376)
Training: (nnz = 63, max change = 0.017579)
Training: (nnz = 52, max change = 0.005534)
Training: (nnz = 52, max change = 0.019374)
Training: (nnz = 50, max change = 0.005859)
Training: (nnz = 56, max change = 0.013841)
Training: (nnz = 49, max change = 0.007306)
Training: (nnz = 51, max change = 0.017447)
Training: (nnz = 42, max change = 0.001871)
Training: (nnz = 44, max change = 0.005889)
Training: (nnz = 43, max change = 0.004112)
Training: (nnz = 49, max change = 0.020014)
Training: (nnz = 45, max change = 0.002811)
Training: (nnz = 39, max change = 0.003848)
Training: (nnz = 51, max change = 0.018842)
Training: (nnz = 37, max change = 0.001834)
Training: (nnz = 34, max change = 0.004377)
Training: (nnz = 42, max change = 0.013879)
Training: (nnz = 35, max change = 0.001855)
Training: (nnz = 40, max change = 0.016622)
Training: (nnz = 36, max change = 0.001042)
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)

Mark Schmidt > Software > L1General > Examples of using L1General